Nov 01 05:44:47.821619 ip-10-0-129-213 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 05:44:48.251584 ip-10-0-129-213 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 05:44:48.251584 ip-10-0-129-213 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 05:44:48.251584 ip-10-0-129-213 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 05:44:48.251584 ip-10-0-129-213 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 05:44:48.256777 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256685    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:44:48.256777 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256772    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:44:48.256777 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256778    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:44:48.256777 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256782    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:44:48.256777 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256786    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:44:48.257080 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256790    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:44:48.257080 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256794    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:44:48.257080 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256799    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:44:48.257080 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256802    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:44:48.257080 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256807    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:44:48.257080 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256811    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:44:48.257080 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256815    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:44:48.257080 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256818    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:44:48.257080 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256822    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:44:48.257080 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256826    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:44:48.257080 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256840    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:44:48.257080 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256844    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:44:48.257080 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256849    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:44:48.257080 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256854    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:44:48.257080 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256858    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:44:48.257080 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256862    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:44:48.257080 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256869    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:44:48.257080 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256875    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:44:48.257080 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256880    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:44:48.257911 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256886    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:44:48.257911 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256891    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:44:48.257911 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256895    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:44:48.257911 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256900    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:44:48.257911 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256904    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:44:48.257911 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256908    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:44:48.257911 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256913    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:44:48.257911 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256917    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:44:48.257911 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256921    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:44:48.257911 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256926    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:44:48.257911 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256930    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:44:48.257911 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256934    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:44:48.257911 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256939    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:44:48.257911 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256945    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:44:48.257911 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256949    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:44:48.257911 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256962    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:44:48.257911 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256967    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:44:48.257911 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256971    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:44:48.257911 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256976    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:44:48.258715 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256980    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:44:48.258715 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256984    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:44:48.258715 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256988    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:44:48.258715 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256993    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:44:48.258715 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.256998    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:44:48.258715 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257003    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:44:48.258715 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257007    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:44:48.258715 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257012    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:44:48.258715 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257016    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:44:48.258715 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257021    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:44:48.258715 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257025    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:44:48.258715 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257029    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:44:48.258715 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257033    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:44:48.258715 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257038    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:44:48.258715 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257043    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:44:48.258715 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257046    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:44:48.258715 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257050    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:44:48.258715 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257055    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:44:48.258715 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257059    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:44:48.258715 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257064    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:44:48.259568 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257068    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:44:48.259568 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257072    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:44:48.259568 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257076    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:44:48.259568 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257081    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:44:48.259568 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257085    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:44:48.259568 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257089    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:44:48.259568 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257093    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:44:48.259568 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257098    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:44:48.259568 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257102    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:44:48.259568 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257106    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:44:48.259568 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257111    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:44:48.259568 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257115    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:44:48.259568 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257119    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:44:48.259568 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257123    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:44:48.259568 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257127    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:44:48.259568 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257132    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:44:48.259568 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257136    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:44:48.259568 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257141    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:44:48.259568 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257145    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:44:48.259568 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257149    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:44:48.260455 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257154    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:44:48.260455 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257158    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:44:48.260455 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257162    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:44:48.260455 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257166    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:44:48.260455 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257171    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:44:48.260455 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257175    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:44:48.260455 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257180    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:44:48.260455 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257184    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:44:48.260455 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257188    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:44:48.260455 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257192    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:44:48.260455 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257197    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:44:48.260455 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257202    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:44:48.260455 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257207    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:44:48.260455 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257211    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:44:48.260455 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257216    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:44:48.260455 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257220    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:44:48.260455 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257224    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:44:48.260455 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257231    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:44:48.260455 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257236    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:44:48.260945 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257241    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:44:48.260945 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257245    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:44:48.260945 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257250    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:44:48.260945 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.257254    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:44:48.260945 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258600    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:44:48.260945 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258612    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:44:48.260945 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258617    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:44:48.260945 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258622    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:44:48.260945 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258626    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:44:48.260945 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258630    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:44:48.260945 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258635    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:44:48.260945 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258640    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:44:48.260945 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258649    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:44:48.260945 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258655    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:44:48.260945 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258660    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:44:48.260945 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258664    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:44:48.260945 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258668    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:44:48.260945 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258672    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:44:48.260945 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258677    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:44:48.260945 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258681    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:44:48.261459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258685    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:44:48.261459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258689    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:44:48.261459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258694    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:44:48.261459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258698    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:44:48.261459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258702    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:44:48.261459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258706    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:44:48.261459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258710    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:44:48.261459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258715    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:44:48.261459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258719    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:44:48.261459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258723    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:44:48.261459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258727    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:44:48.261459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258731    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:44:48.261459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258736    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:44:48.261459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258740    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:44:48.261459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258744    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:44:48.261459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258749    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:44:48.261459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258753    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:44:48.261459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258757    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:44:48.261459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258762    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:44:48.261971 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258766    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:44:48.261971 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258770    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:44:48.261971 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258775    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:44:48.261971 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258779    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:44:48.261971 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258783    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:44:48.261971 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258789    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:44:48.261971 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258794    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:44:48.261971 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258798    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:44:48.261971 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258802    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:44:48.261971 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258806    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:44:48.261971 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258810    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:44:48.261971 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258815    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:44:48.261971 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258820    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:44:48.261971 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258823    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:44:48.261971 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258828    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:44:48.261971 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258832    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:44:48.261971 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258836    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:44:48.261971 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258841    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:44:48.261971 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258845    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:44:48.261971 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258850    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:44:48.262846 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258854    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:44:48.262846 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258858    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:44:48.262846 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258862    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:44:48.262846 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258866    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:44:48.262846 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258871    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:44:48.262846 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258875    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:44:48.262846 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258879    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:44:48.262846 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258883    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:44:48.262846 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258888    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:44:48.262846 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258894    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:44:48.262846 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258900    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:44:48.262846 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258906    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:44:48.262846 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258911    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:44:48.262846 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258915    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:44:48.262846 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258920    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:44:48.262846 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258924    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:44:48.262846 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258928    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:44:48.262846 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258933    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:44:48.262846 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258938    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:44:48.262846 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258942    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:44:48.263765 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258946    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:44:48.263765 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258950    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:44:48.263765 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258954    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:44:48.263765 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258958    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:44:48.263765 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258963    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:44:48.263765 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258967    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:44:48.263765 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258971    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:44:48.263765 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258976    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:44:48.263765 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258980    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:44:48.263765 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258984    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:44:48.263765 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258988    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:44:48.263765 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258993    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:44:48.263765 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.258997    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:44:48.263765 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259001    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:44:48.263765 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259006    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:44:48.263765 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259010    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:44:48.263765 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259014    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:44:48.263765 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259018    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:44:48.263765 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259023    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:44:48.264446 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259027    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:44:48.264446 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259031    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:44:48.264446 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259035    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:44:48.264446 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259039    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:44:48.264446 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259045    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:44:48.264446 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259049    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:44:48.264446 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259054    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:44:48.264446 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259058    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:44:48.264446 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259062    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:44:48.264446 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259066    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:44:48.264446 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259070    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:44:48.264446 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259074    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:44:48.264446 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259912    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:44:48.264446 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259922    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:44:48.264446 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259927    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:44:48.264446 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259931    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:44:48.264446 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259936    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:44:48.264446 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259940    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:44:48.264446 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259945    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:44:48.264446 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259950    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:44:48.264920 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259954    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:44:48.264920 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259958    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:44:48.264920 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259962    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:44:48.264920 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259967    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:44:48.264920 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259971    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:44:48.264920 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259975    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:44:48.264920 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259979    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:44:48.264920 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259984    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:44:48.264920 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259988    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:44:48.264920 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259993    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:44:48.264920 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.259997    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:44:48.264920 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260001    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:44:48.264920 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260006    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:44:48.264920 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260010    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:44:48.264920 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260014    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:44:48.264920 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260018    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:44:48.264920 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260022    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:44:48.264920 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260027    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:44:48.264920 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260031    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:44:48.264920 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260035    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:44:48.265412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260039    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:44:48.265412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260044    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:44:48.265412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260048    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:44:48.265412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260052    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:44:48.265412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260057    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:44:48.265412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260061    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:44:48.265412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260066    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:44:48.265412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260070    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:44:48.265412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260074    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:44:48.265412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260079    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:44:48.265412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260082    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:44:48.265412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260087    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:44:48.265412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260091    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:44:48.265412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260095    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:44:48.265412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260099    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:44:48.265412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260103    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:44:48.265412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260108    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:44:48.265412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260112    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:44:48.265412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260117    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:44:48.265412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260121    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:44:48.265890 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260125    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:44:48.265890 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260129    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:44:48.265890 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260133    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:44:48.265890 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260139    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:44:48.265890 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260143    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:44:48.265890 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260147    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:44:48.265890 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260151    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:44:48.265890 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260156    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:44:48.265890 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260160    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:44:48.265890 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260164    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:44:48.265890 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260172    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:44:48.265890 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260178    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:44:48.265890 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260183    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:44:48.265890 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260188    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:44:48.265890 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260192    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:44:48.265890 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260197    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:44:48.265890 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260201    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:44:48.265890 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260206    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:44:48.265890 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260210    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:44:48.266349 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260214    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:44:48.266349 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260219    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:44:48.266349 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260223    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:44:48.266349 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260227    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:44:48.266349 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260231    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:44:48.266349 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260236    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:44:48.266349 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260240    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:44:48.266349 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260244    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:44:48.266349 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260248    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:44:48.266349 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260255    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:44:48.266349 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260259    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:44:48.266349 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260263    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:44:48.266349 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260268    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:44:48.266349 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260272    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:44:48.266349 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260276    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:44:48.266349 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260280    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:44:48.266349 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260285    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:44:48.266349 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260290    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:44:48.266349 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260294    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:44:48.266349 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260298    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:44:48.266888 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260302    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:44:48.266888 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260306    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:44:48.266888 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260310    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:44:48.266888 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260314    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:44:48.266888 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260335    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:44:48.266888 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260340    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:44:48.266888 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260344    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:44:48.266888 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260348    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:44:48.266888 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260352    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:44:48.266888 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260357    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:44:48.266888 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260363    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:44:48.266888 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260370    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:44:48.266888 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260374    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:44:48.266888 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260379    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:44:48.266888 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260383    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:44:48.266888 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260388    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:44:48.266888 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260392    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:44:48.266888 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260398    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:44:48.266888 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.260403    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:44:48.266888 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261854    2583 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261870    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261880    2583 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261887    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261895    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261900    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261907    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261914    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261919    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261924    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261929    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261935    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261941    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261946    2583 flags.go:64] FLAG: --cgroup-root=""
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261950    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261955    2583 flags.go:64] FLAG: --client-ca-file=""
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261959    2583 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261964    2583 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261972    2583 flags.go:64] FLAG: --cluster-domain=""
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261977    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261981    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261986    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261991    2583 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.261998    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 05:44:48.267367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262003    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262009    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262015    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262020    2583 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262024    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262029    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262034    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262039    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262046    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262051    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262056    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262061    2583 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262066    2583 flags.go:64] FLAG: --enable-server="true"
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262071    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262077    2583 flags.go:64] FLAG: --event-burst="100"
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262082    2583 flags.go:64] FLAG: --event-qps="50"
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262087    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262092    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262097    2583 flags.go:64] FLAG: --eviction-hard=""
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262104    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262109    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262114    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262119    2583 flags.go:64] FLAG: --eviction-soft=""
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262123    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262129    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 05:44:48.268018 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262133    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262138    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262142    2583 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262147    2583 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262155    2583 flags.go:64] FLAG: --feature-gates=""
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262162    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262167    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262172    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262177    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262183    2583 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262188    2583 flags.go:64] FLAG: --help="false"
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262193    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262198    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262203    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262207    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262214    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262220    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262224    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262229    2583 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262234    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262239    2583 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262244    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262249    2583 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262253    2583 flags.go:64] FLAG: --kube-reserved=""
Nov 01 05:44:48.268655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262258    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262263    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262268    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262273    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262277    2583 flags.go:64] FLAG: --lock-file=""
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262282    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262287    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262292    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262301    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262306    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262311    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262316    2583 flags.go:64] FLAG: --logging-format="text"
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262337    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262343    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262348    2583 flags.go:64] FLAG: --manifest-url=""
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262352    2583 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262360    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262365    2583 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262373    2583 flags.go:64] FLAG: --max-pods="110"
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262378    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262383    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262388    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262392    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262397    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262402    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 05:44:48.269249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262406    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262417    2583 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262422    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262427    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262432    2583 flags.go:64] FLAG: --pod-cidr=""
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262436    2583 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262440    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262445    2583 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262450    2583 flags.go:64] FLAG: --port="10250"
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262455    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262460    2583 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-000ff5393d2565801"
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262465    2583 flags.go:64] FLAG: --qos-reserved=""
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262470    2583 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262474    2583 flags.go:64] FLAG: --register-node="true"
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262479    2583 flags.go:64] FLAG: --register-with-taints=""
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262485    2583 flags.go:64] FLAG: --registry-burst="10"
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262490    2583 flags.go:64] FLAG: --registry-qps="5"
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262495    2583 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262499    2583 flags.go:64] FLAG: --reserved-memory=""
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262507    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262513    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262518    2583 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262522    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262527    2583 flags.go:64] FLAG: --runonce="false"
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262532    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262537    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262543    2583 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 05:44:48.269947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262550    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262555    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262560    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262565    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262571    2583 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262576    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262581    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262585    2583 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262590    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262595    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262600    2583 flags.go:64] FLAG: --system-cgroups=""
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262604    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262613    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262618    2583 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262622    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262666    2583 flags.go:64] FLAG: --tls-min-version=""
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262672    2583 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262677    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262681    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262686    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262691    2583 flags.go:64] FLAG: --v="2"
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262698    2583 flags.go:64] FLAG: --version="false"
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262705    2583 flags.go:64] FLAG: --vmodule=""
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262711    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.262718    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 05:44:48.270585 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.262932    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:44:48.271424 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.262941    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:44:48.271424 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.262949    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:44:48.271424 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.262953    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:44:48.271424 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.262958    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:44:48.271424 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.262962    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:44:48.271424 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.262967    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:44:48.271424 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.262974    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:44:48.271424 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.262979    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:44:48.271424 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.262983    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:44:48.271424 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.262987    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:44:48.271424 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.262991    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:44:48.271424 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.262996    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:44:48.271424 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263000    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:44:48.271424 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263005    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:44:48.271424 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263010    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:44:48.271424 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263014    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:44:48.271424 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263018    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:44:48.271424 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263023    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:44:48.271424 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263027    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:44:48.272223 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263031    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:44:48.272223 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263035    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:44:48.272223 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263040    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:44:48.272223 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263044    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:44:48.272223 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263049    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:44:48.272223 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263053    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:44:48.272223 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263057    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:44:48.272223 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263061    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:44:48.272223 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263066    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:44:48.272223 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263070    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:44:48.272223 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263074    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:44:48.272223 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263080    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:44:48.272223 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263084    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:44:48.272223 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263089    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:44:48.272223 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263093    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:44:48.272223 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263097    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:44:48.272223 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263101    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:44:48.272223 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263106    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:44:48.272223 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263110    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:44:48.272223 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263115    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:44:48.273077 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263120    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:44:48.273077 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263125    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:44:48.273077 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263129    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:44:48.273077 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263134    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:44:48.273077 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263139    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:44:48.273077 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263143    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:44:48.273077 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263147    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:44:48.273077 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263151    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:44:48.273077 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263156    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:44:48.273077 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263160    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:44:48.273077 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263164    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:44:48.273077 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263168    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:44:48.273077 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263172    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:44:48.273077 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263177    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:44:48.273077 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263181    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:44:48.273077 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263186    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:44:48.273077 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263190    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:44:48.273077 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263194    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:44:48.273077 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263198    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:44:48.273077 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263202    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:44:48.273748 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263207    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:44:48.273748 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263211    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:44:48.273748 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263215    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:44:48.273748 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263221    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:44:48.273748 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263228    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:44:48.273748 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263234    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:44:48.273748 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263240    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:44:48.273748 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263244    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:44:48.273748 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263248    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:44:48.273748 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263253    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:44:48.273748 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263257    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:44:48.273748 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263263    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:44:48.273748 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263267    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:44:48.273748 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263272    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:44:48.273748 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263277    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:44:48.273748 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263281    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:44:48.273748 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263286    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:44:48.273748 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263290    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:44:48.273748 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263295    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:44:48.274282 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263299    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:44:48.274282 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263303    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:44:48.274282 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263308    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:44:48.274282 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263312    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:44:48.274282 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263317    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:44:48.274282 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263337    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:44:48.274282 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263341    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:44:48.274282 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263345    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:44:48.274282 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263349    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:44:48.274282 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263354    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:44:48.274282 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263357    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:44:48.274282 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263362    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:44:48.274282 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263366    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:44:48.274282 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263371    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:44:48.274282 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263375    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:44:48.274282 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263379    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:44:48.274282 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263385    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:44:48.274282 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263390    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:44:48.274282 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263394    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:44:48.275105 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263398    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:44:48.275105 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263402    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:44:48.275105 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263407    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:44:48.275105 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263411    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:44:48.275105 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263416    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:44:48.275105 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263422    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:44:48.275105 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263426    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:44:48.275105 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263430    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:44:48.275105 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.263473    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 05:44:48.275105 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263552    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:44:48.275105 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263560    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:44:48.275105 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263566    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:44:48.275105 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263570    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:44:48.275105 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263576    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:44:48.275105 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263580    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:44:48.275105 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263585    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:44:48.275819 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263589    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:44:48.275819 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263594    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:44:48.275819 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263598    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:44:48.275819 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263602    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:44:48.275819 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263606    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:44:48.275819 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263611    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:44:48.275819 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263618    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:44:48.275819 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263625    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:44:48.275819 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263632    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:44:48.275819 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263637    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:44:48.275819 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263641    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:44:48.275819 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263646    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:44:48.275819 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263651    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:44:48.275819 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263659    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:44:48.275819 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263663    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:44:48.275819 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263668    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:44:48.275819 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263672    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:44:48.275819 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263677    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:44:48.275819 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263681    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:44:48.276536 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263685    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:44:48.276536 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263690    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:44:48.276536 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263694    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:44:48.276536 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263700    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:44:48.276536 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263705    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:44:48.276536 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263709    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:44:48.276536 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263718    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:44:48.276536 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263722    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:44:48.276536 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263726    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:44:48.276536 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263731    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:44:48.276536 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263735    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:44:48.276536 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263740    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:44:48.276536 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263744    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:44:48.276536 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263749    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:44:48.276536 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263754    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:44:48.276536 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263758    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:44:48.276536 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263762    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:44:48.276536 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263766    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:44:48.276536 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263770    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:44:48.277123 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263774    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:44:48.277123 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263778    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:44:48.277123 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263782    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:44:48.277123 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263786    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:44:48.277123 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263790    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:44:48.277123 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263795    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:44:48.277123 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263799    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:44:48.277123 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263804    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:44:48.277123 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263808    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:44:48.277123 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263813    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:44:48.277123 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263817    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:44:48.277123 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263824    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:44:48.277123 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263830    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:44:48.277123 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263835    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:44:48.277123 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263840    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:44:48.277123 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263844    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:44:48.277123 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263851    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:44:48.277123 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263855    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:44:48.277123 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263859    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:44:48.277965 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263863    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:44:48.277965 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263868    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:44:48.277965 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263872    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:44:48.277965 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263877    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:44:48.277965 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263881    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:44:48.277965 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263885    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:44:48.277965 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263890    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:44:48.277965 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263894    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:44:48.277965 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263899    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:44:48.277965 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263903    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:44:48.277965 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263907    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:44:48.277965 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263911    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:44:48.277965 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263916    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:44:48.277965 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263920    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:44:48.277965 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263925    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:44:48.277965 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263929    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:44:48.277965 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263933    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:44:48.277965 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263937    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:44:48.277965 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263942    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:44:48.277965 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263948    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:44:48.278683 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263952    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:44:48.278683 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263956    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:44:48.278683 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263960    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:44:48.278683 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263964    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:44:48.278683 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263969    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:44:48.278683 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263973    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:44:48.278683 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263977    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:44:48.278683 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263982    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:44:48.278683 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263986    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:44:48.278683 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263992    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:44:48.278683 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.263996    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:44:48.278683 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.264001    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:44:48.278683 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.264005    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:44:48.278683 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.264009    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:44:48.278683 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.264014    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:44:48.278683 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.264018    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:44:48.278683 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.264023    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:44:48.278683 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.264027    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:44:48.278683 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.264031    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:44:48.278683 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.264035    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:44:48.279180 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.264040    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:44:48.279180 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.264044    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:44:48.279180 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.273289    2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 05:44:48.279180 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.273487    2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 05:44:48.279180 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.274212    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 05:44:48.279180 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.274223    2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 05:44:48.279180 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274309    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:44:48.279180 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274317    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:44:48.279180 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274342    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:44:48.279180 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274347    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:44:48.279180 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274351    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:44:48.279180 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274356    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:44:48.279180 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274360    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:44:48.279180 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274365    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:44:48.279180 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274370    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:44:48.279180 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274374    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:44:48.279180 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274378    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:44:48.279180 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274382    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:44:48.279180 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274387    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:44:48.279674 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274391    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:44:48.279674 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274395    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:44:48.279674 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274400    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:44:48.279674 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274405    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:44:48.279674 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274409    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:44:48.279674 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274413    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:44:48.279674 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274418    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:44:48.279674 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274424    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:44:48.279674 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274429    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:44:48.279674 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274433    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:44:48.279674 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274438    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:44:48.279674 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274442    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:44:48.279674 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274446    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:44:48.279674 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274450    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:44:48.279674 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274454    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:44:48.279674 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274458    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:44:48.279674 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274462    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:44:48.279674 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274466    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:44:48.279674 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274470    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:44:48.279674 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274474    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:44:48.280138 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274479    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:44:48.280138 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274483    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:44:48.280138 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274487    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:44:48.280138 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274493    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:44:48.280138 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274499    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:44:48.280138 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274504    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:44:48.280138 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274509    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:44:48.280138 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274513    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:44:48.280138 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274518    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:44:48.280138 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274522    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:44:48.280138 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274527    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:44:48.280138 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274531    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:44:48.280138 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274535    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:44:48.280138 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274539    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:44:48.280138 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274543    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:44:48.280138 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274548    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:44:48.280138 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274553    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:44:48.280138 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274557    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:44:48.280138 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274561    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:44:48.280604 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274565    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:44:48.280604 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274571    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:44:48.280604 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274575    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:44:48.280604 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274580    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:44:48.280604 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274584    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:44:48.280604 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274588    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:44:48.280604 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274593    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:44:48.280604 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274597    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:44:48.280604 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274601    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:44:48.280604 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274606    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:44:48.280604 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274610    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:44:48.280604 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274614    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:44:48.280604 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274619    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:44:48.280604 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274623    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:44:48.280604 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274628    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:44:48.280604 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274632    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:44:48.280604 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274636    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:44:48.280604 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274640    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:44:48.280604 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274648    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:44:48.281045 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274654    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:44:48.281045 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274661    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:44:48.281045 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274667    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:44:48.281045 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274672    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:44:48.281045 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274676    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:44:48.281045 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274681    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:44:48.281045 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274685    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:44:48.281045 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274690    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:44:48.281045 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274694    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:44:48.281045 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274698    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:44:48.281045 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274702    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:44:48.281045 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274706    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:44:48.281045 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274711    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:44:48.281045 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274715    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:44:48.281045 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274720    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:44:48.281045 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274725    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:44:48.281045 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274729    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:44:48.281045 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274733    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:44:48.281045 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274737    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:44:48.281045 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274742    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:44:48.281578 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274746    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:44:48.281578 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274750    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:44:48.281578 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274754    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:44:48.281578 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274758    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:44:48.281578 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274763    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:44:48.281578 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274767    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:44:48.281578 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274771    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:44:48.281578 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274775    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:44:48.281578 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274779    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:44:48.281578 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274783    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:44:48.281578 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274788    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:44:48.281578 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274792    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:44:48.281578 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274796    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:44:48.281578 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274800    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:44:48.281578 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.274805    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:44:48.281578 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.274847    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 05:44:48.281968 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275067    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:44:48.281968 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275075    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:44:48.281968 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275080    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:44:48.281968 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275084    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:44:48.281968 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275089    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:44:48.281968 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275093    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:44:48.281968 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275098    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:44:48.281968 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275102    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:44:48.281968 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275108    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:44:48.281968 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275114    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:44:48.281968 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275118    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:44:48.281968 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275123    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:44:48.281968 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275130    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:44:48.281968 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275134    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:44:48.281968 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275139    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:44:48.281968 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275143    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:44:48.281968 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275148    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:44:48.281968 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275152    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:44:48.281968 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275156    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:44:48.281968 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275160    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:44:48.282459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275165    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:44:48.282459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275169    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:44:48.282459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275173    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:44:48.282459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275177    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:44:48.282459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275182    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:44:48.282459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275186    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:44:48.282459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275190    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:44:48.282459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275194    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:44:48.282459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275199    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:44:48.282459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275203    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:44:48.282459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275207    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:44:48.282459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275211    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:44:48.282459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275215    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:44:48.282459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275219    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:44:48.282459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275223    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:44:48.282459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275228    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:44:48.282459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275234    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:44:48.282459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275240    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:44:48.282459 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275244    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:44:48.282918 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275249    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:44:48.282918 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275253    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:44:48.282918 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275257    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:44:48.282918 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275262    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:44:48.282918 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275266    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:44:48.282918 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275270    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:44:48.282918 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275275    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:44:48.282918 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275279    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:44:48.282918 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275283    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:44:48.282918 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275288    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:44:48.282918 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275292    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:44:48.282918 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275296    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:44:48.282918 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275300    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:44:48.282918 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275304    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:44:48.282918 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275309    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:44:48.282918 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275313    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:44:48.282918 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275317    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:44:48.282918 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275337    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:44:48.282918 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275341    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:44:48.283432 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275345    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:44:48.283432 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275349    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:44:48.283432 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275354    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:44:48.283432 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275358    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:44:48.283432 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275362    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:44:48.283432 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275366    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:44:48.283432 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275371    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:44:48.283432 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275375    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:44:48.283432 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275380    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:44:48.283432 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275384    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:44:48.283432 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275388    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:44:48.283432 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275392    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:44:48.283432 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275396    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:44:48.283432 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275400    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:44:48.283432 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275404    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:44:48.283432 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275408    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:44:48.283432 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275413    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:44:48.283432 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275417    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:44:48.283432 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275421    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:44:48.283432 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275425    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:44:48.283944 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275430    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:44:48.283944 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275434    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:44:48.283944 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275438    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:44:48.283944 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275443    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:44:48.283944 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275447    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:44:48.283944 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275451    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:44:48.283944 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275455    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:44:48.283944 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275460    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:44:48.283944 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275464    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:44:48.283944 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275468    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:44:48.283944 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275472    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:44:48.283944 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275476    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:44:48.283944 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275480    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:44:48.283944 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275484    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:44:48.283944 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275489    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:44:48.283944 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275493    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:44:48.283944 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275497    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:44:48.283944 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275504    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:44:48.283944 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275508    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:44:48.284412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275512    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:44:48.284412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275517    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:44:48.284412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275521    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:44:48.284412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275526    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:44:48.284412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275530    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:44:48.284412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275533    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:44:48.284412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275538    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:44:48.284412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275543    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:44:48.284412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275547    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:44:48.284412 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.275587    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 05:44:48.284412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275642    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:44:48.284412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275649    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:44:48.284412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275654    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:44:48.284412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275659    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:44:48.284412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275663    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:44:48.284412 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275668    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:44:48.284792 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275672    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:44:48.284792 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275676    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:44:48.284792 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275680    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:44:48.284792 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275684    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:44:48.284792 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275689    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:44:48.284792 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275694    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:44:48.284792 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275698    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:44:48.284792 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275702    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:44:48.284792 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275706    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:44:48.284792 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275710    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:44:48.284792 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275715    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:44:48.284792 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275719    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:44:48.284792 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275723    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:44:48.284792 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275727    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:44:48.284792 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275731    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:44:48.284792 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275739    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:44:48.284792 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275743    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:44:48.284792 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275748    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:44:48.284792 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275752    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:44:48.284792 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275756    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:44:48.285250 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275760    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:44:48.285250 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275765    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:44:48.285250 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275769    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:44:48.285250 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275773    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:44:48.285250 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275779    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:44:48.285250 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275784    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:44:48.285250 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275789    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:44:48.285250 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275793    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:44:48.285250 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275797    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:44:48.285250 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275802    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:44:48.285250 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275806    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:44:48.285250 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275810    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:44:48.285250 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275815    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:44:48.285250 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275819    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:44:48.285250 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275823    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:44:48.285250 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275827    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:44:48.285250 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275831    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:44:48.285250 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275835    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:44:48.285250 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275839    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:44:48.285714 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275843    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:44:48.285714 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275848    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:44:48.285714 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275852    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:44:48.285714 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275856    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:44:48.285714 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275861    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:44:48.285714 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275865    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:44:48.285714 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275870    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:44:48.285714 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275874    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:44:48.285714 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275882    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:44:48.285714 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275886    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:44:48.285714 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275890    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:44:48.285714 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275895    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:44:48.285714 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275900    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:44:48.285714 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275905    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:44:48.285714 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275909    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:44:48.285714 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275916    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:44:48.285714 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275923    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:44:48.285714 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275930    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:44:48.285714 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275935    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:44:48.285714 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275941    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:44:48.286188 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275945    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:44:48.286188 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275950    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:44:48.286188 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275954    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:44:48.286188 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275959    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:44:48.286188 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275963    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:44:48.286188 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275968    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:44:48.286188 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275972    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:44:48.286188 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275976    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:44:48.286188 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275981    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:44:48.286188 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275985    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:44:48.286188 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275989    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:44:48.286188 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275993    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:44:48.286188 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.275997    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:44:48.286188 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276001    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:44:48.286188 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276005    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:44:48.286188 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276009    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:44:48.286188 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276014    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:44:48.286188 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276018    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:44:48.286188 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276022    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:44:48.286188 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276026    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:44:48.286679 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276032    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:44:48.286679 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276036    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:44:48.286679 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276040    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:44:48.286679 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276046    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:44:48.286679 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276053    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:44:48.286679 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276058    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:44:48.286679 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276062    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:44:48.286679 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276066    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:44:48.286679 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276070    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:44:48.286679 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276076    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:44:48.286679 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276081    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:44:48.286679 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276085    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:44:48.286679 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276089    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:44:48.286679 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276093    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:44:48.286679 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276097    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:44:48.286679 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276103    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:44:48.286679 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276107    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:44:48.286679 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276111    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:44:48.286679 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276116    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:44:48.287130 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276120    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:44:48.287130 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:48.276124    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:44:48.287130 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.276197    2583 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 05:44:48.287130 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.279108    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 01 05:44:48.287130 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.281479    2583 server.go:1014] "Starting client certificate rotation"
Nov 01 05:44:48.287130 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.281576    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 05:44:48.287130 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.282550    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 05:44:48.287130 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.283839    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 05:44:48.287130 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.285116    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 05:44:48.297808 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.297791    2583 log.go:25] "Validated CRI v1 runtime API"
Nov 01 05:44:48.302138 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.302124    2583 log.go:25] "Validated CRI v1 image API"
Nov 01 05:44:48.302995 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.302971    2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 05:44:48.305265 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.305246    2583 fs.go:135] Filesystem UUIDs: map[47b6056d-103c-4176-b5e0-12d2c1a2326b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9c6745f8-50ec-4200-9c72-eea0bd7cf50a:/dev/nvme0n1p3]
Nov 01 05:44:48.305364 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.305264    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 01 05:44:48.310255 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.310146    2583 manager.go:217] Machine: {Timestamp:2026-11-01 05:44:48.308823952 +0000 UTC m=+0.365426445 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098630 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c281b553c640ecb4e199644f9666b SystemUUID:ec2c281b-553c-640e-cb4e-199644f9666b BootID:5c4d2b0c-4260-4bf2-8720-61ca0aaf76de Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4f:20:67:74:0b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4f:20:67:74:0b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:58:05:4a:01:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 05:44:48.310722 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.310712    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 05:44:48.310824 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.310813    2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 05:44:48.313611 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.313587    2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 05:44:48.313781 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.313615    2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-213.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 05:44:48.313913 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.313785    2583 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 05:44:48.313913 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.313799    2583 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 05:44:48.313913 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.313849    2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 05:44:48.313913 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.313874    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 05:44:48.314658 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.314644    2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 05:44:48.318410 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.318396    2583 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 05:44:48.318544 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.318532    2583 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 05:44:48.319147 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.319136    2583 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 05:44:48.319203 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.319155    2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 05:44:48.319203 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.319170    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 05:44:48.319203 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.319183    2583 kubelet.go:405] "Adding apiserver pod source"
Nov 01 05:44:48.319203 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.319195    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 05:44:48.319611 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.319594    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 05:44:48.323334 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.323285    2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 05:44:48.328343 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.328313    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 05:44:48.329102 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.329087    2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 05:44:48.329176 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.329129    2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 05:44:48.329348 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.329334    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 05:44:48.329417 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.329357    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 05:44:48.329417 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.329367    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 05:44:48.329417 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.329377    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 05:44:48.329417 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.329386    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 05:44:48.329417 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.329397    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 05:44:48.329853 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.329842    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 05:44:48.329901 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.329858    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 05:44:48.329901 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.329869    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 05:44:48.329901 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.329880    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 05:44:48.329901 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.329896    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 05:44:48.330107 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.329911    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 05:44:48.330508 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.330496    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 05:44:48.330559 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.330512    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 05:44:48.334673 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.334657    2583 server.go:1263] "Started kubelet"
Nov 01 05:44:48.334763 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.334731    2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 05:44:48.334856 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.334808    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 05:44:48.334887 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.334874    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 05:44:48.335881 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.335865    2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 05:44:48.335972 ip-10-0-129-213 systemd[1]: Started Kubernetes Kubelet.
Nov 01 05:44:48.338670 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.337850    2583 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 05:44:48.340560 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:48.340544    2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 01 05:44:48.342273 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.342246    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 05:44:48.343088 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.343070    2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 05:44:48.343219 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:48.343189    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-213.us-west-1.compute.internal\" not found"
Nov 01 05:44:48.343599 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.343583    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 05:44:48.343599 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.343586    2583 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 05:44:48.343730 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.343610    2583 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 05:44:48.343730 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.343684    2583 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 05:44:48.343730 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.343693    2583 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 05:44:48.343895 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.343873    2583 factory.go:55] Registering systemd factory
Nov 01 05:44:48.343895 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.343896    2583 factory.go:223] Registration of the systemd container factory successfully
Nov 01 05:44:48.344130 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.344118    2583 factory.go:153] Registering CRI-O factory
Nov 01 05:44:48.344130 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.344132    2583 factory.go:223] Registration of the crio container factory successfully
Nov 01 05:44:48.344202 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.344176    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 05:44:48.344202 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.344192    2583 factory.go:103] Registering Raw factory
Nov 01 05:44:48.344285 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.344205    2583 manager.go:1199] Started watching for new ooms in manager
Nov 01 05:44:48.345019 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.345006    2583 manager.go:319] Starting recovery of all containers
Nov 01 05:44:48.350483 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.350455    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 05:44:48.356560 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.356540    2583 manager.go:324] Recovery completed
Nov 01 05:44:48.362714 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.362698    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:44:48.364378 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.364348    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:44:48.364499 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.364399    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:44:48.364499 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.364416    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:44:48.365033 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.365018    2583 cpu_manager.go:226] "Starting" policy="none"
Nov 01 05:44:48.365087 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.365032    2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 05:44:48.365087 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.365041    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:44:48.365087 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.365054    2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 05:44:48.365843 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.365826    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:44:48.365921 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.365859    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:44:48.365921 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.365875    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:44:48.367282 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.367269    2583 policy_none.go:50] "Start"
Nov 01 05:44:48.367347 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.367285    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 05:44:48.367347 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.367295    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 05:44:48.369294 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.369284    2583 policy_none.go:44] "Start"
Nov 01 05:44:48.412006 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.411966    2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 05:44:48.414479 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.414463    2583 manager.go:341] "Starting Device Plugin manager"
Nov 01 05:44:48.414580 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:48.414493    2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 05:44:48.414580 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.414502    2583 server.go:87] "Starting device plugin registration server"
Nov 01 05:44:48.414883 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.414865    2583 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 05:44:48.414883 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.414881    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 05:44:48.416032 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.415132    2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 05:44:48.416032 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.415229    2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 05:44:48.416032 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.415238    2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 05:44:48.436692 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:48.417483    2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 05:44:48.436692 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:48.417532    2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-213.us-west-1.compute.internal\" not found"
Nov 01 05:44:48.464019 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.463992    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 05:44:48.464019 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.464019    2583 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 05:44:48.464169 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.464037    2583 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 05:44:48.464169 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:48.464066    2583 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 01 05:44:48.481335 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.481303    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-213.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 01 05:44:48.481335 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:48.481306    2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-213.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 01 05:44:48.482492 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:48.481296    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-213.us-west-1.compute.internal.18e3d58f5ca029d6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-213.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-213.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 05:44:48.334629334 +0000 UTC m=+0.391231827,LastTimestamp:2026-11-01 05:44:48.334629334 +0000 UTC m=+0.391231827,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-213.us-west-1.compute.internal,}"
Nov 01 05:44:48.515183 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.515104    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:44:48.516010 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.515995    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:44:48.516093 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.516025    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:44:48.516093 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.516037    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:44:48.516093 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.516083    2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:44:48.564959 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.564935    2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal"]
Nov 01 05:44:48.565086 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.565001    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:44:48.565197 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.565181    2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 05:44:48.565250 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.565227    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-597pc"
Nov 01 05:44:48.565947 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.565932    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:44:48.566026 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.565986    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:44:48.566026 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.565999    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:44:48.568195 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.568181    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:44:48.568353 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.568339    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:44:48.568392 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.568371    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:44:48.568930 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.568915    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:44:48.569004 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.568946    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:44:48.569004 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.568915    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:44:48.569004 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.568962    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:44:48.569004 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.568975    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:44:48.569004 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.568988    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:44:48.571105 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.571090    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:44:48.571173 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.571128    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:44:48.571829 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.571812    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:44:48.571895 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.571845    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:44:48.571895 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.571860    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:44:48.601954 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:48.601937    2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-213.us-west-1.compute.internal\" not found" node="ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:44:48.608181 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:48.608165    2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-213.us-west-1.compute.internal\" not found" node="ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:44:48.642387 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.642368    2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:44:48.645588 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.645569    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef0f634a7979920e9df606f3bb474aac-config\") pod \"kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal\" (UID: \"ef0f634a7979920e9df606f3bb474aac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:44:48.645674 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.645597    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28a06cb4c587db3aac34f3d8afd3bbd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal\" (UID: \"28a06cb4c587db3aac34f3d8afd3bbd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:44:48.645674 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.645614    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28a06cb4c587db3aac34f3d8afd3bbd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal\" (UID: \"28a06cb4c587db3aac34f3d8afd3bbd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:44:48.690390 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.690362    2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:44:48.740357 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.740335    2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:44:48.741497 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.741476    2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:44:48.741693 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.741679    2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:44:48.741760 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:48.741705    2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-213.us-west-1.compute.internal\": node \"ip-10-0-129-213.us-west-1.compute.internal\" not found"
Nov 01 05:44:48.744010 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.743997    2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:44:48.746176 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.746160    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef0f634a7979920e9df606f3bb474aac-config\") pod \"kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal\" (UID: \"ef0f634a7979920e9df606f3bb474aac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:44:48.746245 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.746189    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28a06cb4c587db3aac34f3d8afd3bbd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal\" (UID: \"28a06cb4c587db3aac34f3d8afd3bbd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:44:48.746245 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.746215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28a06cb4c587db3aac34f3d8afd3bbd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal\" (UID: \"28a06cb4c587db3aac34f3d8afd3bbd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:44:48.746344 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.746253    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28a06cb4c587db3aac34f3d8afd3bbd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal\" (UID: \"28a06cb4c587db3aac34f3d8afd3bbd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:44:48.746344 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.746257    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef0f634a7979920e9df606f3bb474aac-config\") pod \"kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal\" (UID: \"ef0f634a7979920e9df606f3bb474aac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:44:48.746344 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.746257    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28a06cb4c587db3aac34f3d8afd3bbd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal\" (UID: \"28a06cb4c587db3aac34f3d8afd3bbd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:44:48.806517 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.806468    2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 05:44:48.806851 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.806811    2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:44:48.864310 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.864292    2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 05:44:48.905446 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.905424    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:44:48.912987 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:48.912974    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:44:49.279767 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.279730    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 01 05:44:49.280307 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.279868    2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 05:44:49.280307 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.279919    2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 05:44:49.280307 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.279922    2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 05:44:49.280307 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:49.279952    2583 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-129-213.us-west-1.compute.internal.18e3d58f5e66c7a8\": read tcp 10.0.129.213:60920->18.189.164.78:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-129-213.us-west-1.compute.internal.18e3d58f5e66c7a8  default   5092 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-213.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-213.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-213.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 05:44:48 +0000 UTC,LastTimestamp:2026-11-01 05:44:48.51604358 +0000 UTC m=+0.572646074,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-213.us-west-1.compute.internal,}"
Nov 01 05:44:49.320072 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.320047    2583 apiserver.go:52] "Watching apiserver"
Nov 01 05:44:49.343463 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.343441    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 01 05:44:49.570407 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.570304    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 05:39:48 +0000 UTC" deadline="2028-08-16 14:56:12.951977802 +0000 UTC"
Nov 01 05:44:49.570407 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.570352    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15705h11m23.381628613s"
Nov 01 05:44:49.611788 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.611755    2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 05:44:49.612132 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.612108    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-jh5gk","kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal","openshift-network-operator/iptables-alerter-xv89g","openshift-ovn-kubernetes/ovnkube-node-nt68j","kube-system/konnectivity-agent-4lm7r","openshift-cluster-node-tuning-operator/tuned-z29gq","openshift-multus/multus-additional-cni-plugins-h4ps2","openshift-network-diagnostics/network-check-target-cwn9s","openshift-dns/node-resolver-xqcsn","openshift-image-registry/node-ca-dnkvk","openshift-multus/multus-7xz4d","openshift-multus/network-metrics-daemon-gjh6r"]
Nov 01 05:44:49.614803 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.614783    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:44:49.614910 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:49.614856    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jh5gk" podUID="7f767e63-7cc7-4b5e-9941-860fe5f4cd62"
Nov 01 05:44:49.617055 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.617038    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.619457 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.619438    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xv89g"
Nov 01 05:44:49.621850 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.621834    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.624184 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.624169    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4lm7r"
Nov 01 05:44:49.626501 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.626481    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.628820 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.628804    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:49.631094 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.631080    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:44:49.631177 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:49.631159    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cwn9s" podUID="a78d89dc-2ed6-4a2e-9558-f54b73a806d2"
Nov 01 05:44:49.633422 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.633403    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xqcsn"
Nov 01 05:44:49.635793 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.635776    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dnkvk"
Nov 01 05:44:49.638185 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.638162    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.640514 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.640496    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:44:49.640605 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:49.640561    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjh6r" podUID="a18b7694-133e-4663-b8f6-a36f981ab717"
Nov 01 05:44:49.644884 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.644866    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 05:44:49.652764 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.652741    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9efa148c-f7c3-4f9d-bee3-cc1eb353b808-iptables-alerter-script\") pod \"iptables-alerter-xv89g\" (UID: \"9efa148c-f7c3-4f9d-bee3-cc1eb353b808\") " pod="openshift-network-operator/iptables-alerter-xv89g"
Nov 01 05:44:49.652854 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.652782    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-run-openvswitch\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.652854 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.652809    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c1d9c47e-944e-4112-9889-b01892353659-ovnkube-config\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.652953 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.652881    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsndz\" (UniqueName: \"kubernetes.io/projected/1e9a8c0a-32d6-44ef-8a47-f65f6560193d-kube-api-access-tsndz\") pod \"node-resolver-xqcsn\" (UID: \"1e9a8c0a-32d6-44ef-8a47-f65f6560193d\") " pod="openshift-dns/node-resolver-xqcsn"
Nov 01 05:44:49.652953 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.652924    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-cni-dir\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.653025 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.652949    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-run-netns\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.653025 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.652986    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5ace06a-0f90-46ef-8a04-fdc0f7180a1e-serviceca\") pod \"node-ca-dnkvk\" (UID: \"e5ace06a-0f90-46ef-8a04-fdc0f7180a1e\") " pod="openshift-image-registry/node-ca-dnkvk"
Nov 01 05:44:49.653100 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653023    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-conf-dir\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.653100 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653058    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-daemon-config\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.653100 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653091    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg2zd\" (UniqueName: \"kubernetes.io/projected/2980ba94-fa03-419a-82d7-86405b6178ac-kube-api-access-vg2zd\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:49.653204 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653115    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-sys\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.653204 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653138    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-run-ovn-kubernetes\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.653204 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653154    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-kubernetes\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.653204 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653176    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-cnibin\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.653423 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653219    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-cni-bin\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.653423 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653283    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1d9c47e-944e-4112-9889-b01892353659-ovn-node-metrics-cert\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.653423 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653321    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2980ba94-fa03-419a-82d7-86405b6178ac-cnibin\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:49.653423 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2980ba94-fa03-419a-82d7-86405b6178ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:49.653423 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653393    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-kubelet-config\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:44:49.653423 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653410    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9efa148c-f7c3-4f9d-bee3-cc1eb353b808-host-slash\") pod \"iptables-alerter-xv89g\" (UID: \"9efa148c-f7c3-4f9d-bee3-cc1eb353b808\") " pod="openshift-network-operator/iptables-alerter-xv89g"
Nov 01 05:44:49.653711 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653426    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-run-netns\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.653711 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653446    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2980ba94-fa03-419a-82d7-86405b6178ac-system-cni-dir\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:49.653711 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653474    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:44:49.653711 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653515    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-var-lib-kubelet\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.653711 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653546    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86v89\" (UniqueName: \"kubernetes.io/projected/d0e384e0-dcf6-410c-b344-8d797c57e1c9-kube-api-access-86v89\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.653711 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653591    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfpq7\" (UniqueName: \"kubernetes.io/projected/e5ace06a-0f90-46ef-8a04-fdc0f7180a1e-kube-api-access-jfpq7\") pod \"node-ca-dnkvk\" (UID: \"e5ace06a-0f90-46ef-8a04-fdc0f7180a1e\") " pod="openshift-image-registry/node-ca-dnkvk"
Nov 01 05:44:49.653711 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653632    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-var-lib-kubelet\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.653711 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653674    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-run-ovn\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.653711 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653692    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-node-log\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.654121 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653733    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-registration-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.654121 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653769    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-etc-selinux\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.654121 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653821    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rprhl\" (UniqueName: \"kubernetes.io/projected/9efa148c-f7c3-4f9d-bee3-cc1eb353b808-kube-api-access-rprhl\") pod \"iptables-alerter-xv89g\" (UID: \"9efa148c-f7c3-4f9d-bee3-cc1eb353b808\") " pod="openshift-network-operator/iptables-alerter-xv89g"
Nov 01 05:44:49.654121 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653875    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-sysconfig\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.654121 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653935    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-hostroot\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.654121 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653953    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-kubelet\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.654121 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.653977    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-log-socket\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.654121 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.654025    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-tuned\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.654121 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.654061    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c1d9c47e-944e-4112-9889-b01892353659-env-overrides\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.654121 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.654099    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-socket-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.654121 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.654125    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e9a8c0a-32d6-44ef-8a47-f65f6560193d-tmp-dir\") pod \"node-resolver-xqcsn\" (UID: \"1e9a8c0a-32d6-44ef-8a47-f65f6560193d\") " pod="openshift-dns/node-resolver-xqcsn"
Nov 01 05:44:49.654556 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.654155    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-socket-dir-parent\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.654556 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.654194    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-var-lib-cni-bin\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.654556 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.654225    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-var-lib-cni-multus\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.654556 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.654260    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-dbus\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:44:49.654556 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.654276    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdeea076-cd20-459e-b1a5-79ec825e95e3-cni-binary-copy\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.654556 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.654320    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-run-multus-certs\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.654849 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.654707    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-device-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.654849 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.654761    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-systemd-units\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.654849 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.654797    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-run-systemd\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.654849 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.654832    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2980ba94-fa03-419a-82d7-86405b6178ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:49.655097 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.654868    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/05e09ea1-324b-49ef-a4a5-2645b50232a4-konnectivity-ca\") pod \"konnectivity-agent-4lm7r\" (UID: \"05e09ea1-324b-49ef-a4a5-2645b50232a4\") " pod="kube-system/konnectivity-agent-4lm7r"
Nov 01 05:44:49.655097 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.654924    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-run\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.655097 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.654987    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-sys-fs\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.655097 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655083    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxjd2\" (UniqueName: \"kubernetes.io/projected/04241194-2fed-4174-9eda-e384dcfd4131-kube-api-access-nxjd2\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.655291 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655145    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-slash\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.655291 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655176    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-etc-openvswitch\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.655291 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655235    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-modprobe-d\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.655291 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655265    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-run-k8s-cni-cncf-io\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.655490 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655301    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-cni-netd\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.655490 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655336    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-systemd\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.655490 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655419    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-host\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.655490 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655455    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-var-lib-openvswitch\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.655676 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655492    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c79lv\" (UniqueName: \"kubernetes.io/projected/c1d9c47e-944e-4112-9889-b01892353659-kube-api-access-c79lv\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.655676 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655520    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs\") pod \"network-metrics-daemon-gjh6r\" (UID: \"a18b7694-133e-4663-b8f6-a36f981ab717\") " pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:44:49.655676 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655555    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwg57\" (UniqueName: \"kubernetes.io/projected/a18b7694-133e-4663-b8f6-a36f981ab717-kube-api-access-kwg57\") pod \"network-metrics-daemon-gjh6r\" (UID: \"a18b7694-133e-4663-b8f6-a36f981ab717\") " pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:44:49.655676 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655589    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/05e09ea1-324b-49ef-a4a5-2645b50232a4-agent-certs\") pod \"konnectivity-agent-4lm7r\" (UID: \"05e09ea1-324b-49ef-a4a5-2645b50232a4\") " pod="kube-system/konnectivity-agent-4lm7r"
Nov 01 05:44:49.655676 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655621    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-sysctl-conf\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.655676 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655658    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-lib-modules\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.655959 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655745    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58hcx\" (UniqueName: \"kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx\") pod \"network-check-target-cwn9s\" (UID: \"a78d89dc-2ed6-4a2e-9558-f54b73a806d2\") " pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:44:49.655959 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655782    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-kubelet-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.655959 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655816    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c1d9c47e-944e-4112-9889-b01892353659-ovnkube-script-lib\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.655959 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655846    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2980ba94-fa03-419a-82d7-86405b6178ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:49.655959 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655880    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5ace06a-0f90-46ef-8a04-fdc0f7180a1e-host\") pod \"node-ca-dnkvk\" (UID: \"e5ace06a-0f90-46ef-8a04-fdc0f7180a1e\") " pod="openshift-image-registry/node-ca-dnkvk"
Nov 01 05:44:49.655959 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655913    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2980ba94-fa03-419a-82d7-86405b6178ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:49.655959 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655945    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-sysctl-d\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.656271 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.655969    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0e384e0-dcf6-410c-b344-8d797c57e1c9-tmp\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.656271 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.656004    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e9a8c0a-32d6-44ef-8a47-f65f6560193d-hosts-file\") pod \"node-resolver-xqcsn\" (UID: \"1e9a8c0a-32d6-44ef-8a47-f65f6560193d\") " pod="openshift-dns/node-resolver-xqcsn"
Nov 01 05:44:49.656271 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.656034    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-system-cni-dir\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.656271 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.656066    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-etc-kubernetes\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.656271 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.656100    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.656271 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.656131    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwxk7\" (UniqueName: \"kubernetes.io/projected/bdeea076-cd20-459e-b1a5-79ec825e95e3-kube-api-access-nwxk7\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.675481 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.675457    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 05:44:49.675624 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.675604    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 05:44:49.675769 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.675754    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 05:44:49.675829 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.675800    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 05:44:49.675829 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.675803    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 05:44:49.675921 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.675847    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jzphk\""
Nov 01 05:44:49.675921 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.675857    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-695z8\""
Nov 01 05:44:49.675921 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.675880    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 05:44:49.677791 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.677770    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 05:44:49.678189 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.678089    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 05:44:49.678189 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.678137    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 05:44:49.678189 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.678152    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vbc6d\""
Nov 01 05:44:49.678189 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.678157    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 05:44:49.678189 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.678167    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 05:44:49.678189 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.678194    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 05:44:49.680087 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.680072    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xlcv8\""
Nov 01 05:44:49.680161 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.680149    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 05:44:49.680161 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.680147    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 05:44:49.721407 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.721387    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 05:44:49.727605 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.727587    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 05:44:49.727605 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.727601    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rpsgx\""
Nov 01 05:44:49.729495 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.729475    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 05:44:49.729585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.729504    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9tjqf\""
Nov 01 05:44:49.729585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.729562    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m2d6d\""
Nov 01 05:44:49.729585 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.729566    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qfzvw\""
Nov 01 05:44:49.731297 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.731282    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 05:44:49.731367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.731307    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 05:44:49.731367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.731343    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 05:44:49.731367 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.731343    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 05:44:49.731612 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.731576    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 05:44:49.731671 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.731625    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 05:44:49.731671 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.731631    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 05:44:49.731764 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.731676    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 05:44:49.731980 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.731948    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rz4ht\""
Nov 01 05:44:49.732092 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.732031    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 05:44:49.732092 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.732034    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 05:44:49.732092 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.732062    2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 05:44:49.732220 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.732102    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pkb92"
Nov 01 05:44:49.757141 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.757122    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwxk7\" (UniqueName: \"kubernetes.io/projected/bdeea076-cd20-459e-b1a5-79ec825e95e3-kube-api-access-nwxk7\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.757229 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.757147    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9efa148c-f7c3-4f9d-bee3-cc1eb353b808-iptables-alerter-script\") pod \"iptables-alerter-xv89g\" (UID: \"9efa148c-f7c3-4f9d-bee3-cc1eb353b808\") " pod="openshift-network-operator/iptables-alerter-xv89g"
Nov 01 05:44:49.757229 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.757165    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-run-openvswitch\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.757229 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.757183    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c1d9c47e-944e-4112-9889-b01892353659-ovnkube-config\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.757229 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.757207    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsndz\" (UniqueName: \"kubernetes.io/projected/1e9a8c0a-32d6-44ef-8a47-f65f6560193d-kube-api-access-tsndz\") pod \"node-resolver-xqcsn\" (UID: \"1e9a8c0a-32d6-44ef-8a47-f65f6560193d\") " pod="openshift-dns/node-resolver-xqcsn"
Nov 01 05:44:49.757447 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.757232    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-cni-dir\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.757447 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.757261    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-run-netns\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.757447 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.757297    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-run-openvswitch\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.757447 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.757318    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-run-netns\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.757447 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.757367    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5ace06a-0f90-46ef-8a04-fdc0f7180a1e-serviceca\") pod \"node-ca-dnkvk\" (UID: \"e5ace06a-0f90-46ef-8a04-fdc0f7180a1e\") " pod="openshift-image-registry/node-ca-dnkvk"
Nov 01 05:44:49.757447 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.757406    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-conf-dir\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.757447 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.757439    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-daemon-config\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.757771 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.757469    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg2zd\" (UniqueName: \"kubernetes.io/projected/2980ba94-fa03-419a-82d7-86405b6178ac-kube-api-access-vg2zd\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:49.757771 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.757476    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-conf-dir\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.757771 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.757498    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-sys\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.757771 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.757592    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-sys\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.757771 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.757734    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-run-ovn-kubernetes\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.757922 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.757822    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c1d9c47e-944e-4112-9889-b01892353659-ovnkube-config\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.757922 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.757851    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5ace06a-0f90-46ef-8a04-fdc0f7180a1e-serviceca\") pod \"node-ca-dnkvk\" (UID: \"e5ace06a-0f90-46ef-8a04-fdc0f7180a1e\") " pod="openshift-image-registry/node-ca-dnkvk"
Nov 01 05:44:49.757984 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.757933    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-cni-dir\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.758016 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.757992    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-daemon-config\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.758240 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.757525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-run-ovn-kubernetes\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.758288 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758273    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-kubernetes\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.758323 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-cnibin\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.758356 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758329    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-cni-bin\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.758393 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758355    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1d9c47e-944e-4112-9889-b01892353659-ovn-node-metrics-cert\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.758393 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758382    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2980ba94-fa03-419a-82d7-86405b6178ac-cnibin\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:49.758502 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758429    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2980ba94-fa03-419a-82d7-86405b6178ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:49.758502 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758457    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-kubelet-config\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:44:49.758502 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758460    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-cnibin\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.758502 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758460    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-cni-bin\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.758502 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758473    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9efa148c-f7c3-4f9d-bee3-cc1eb353b808-iptables-alerter-script\") pod \"iptables-alerter-xv89g\" (UID: \"9efa148c-f7c3-4f9d-bee3-cc1eb353b808\") " pod="openshift-network-operator/iptables-alerter-xv89g"
Nov 01 05:44:49.758710 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758498    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9efa148c-f7c3-4f9d-bee3-cc1eb353b808-host-slash\") pod \"iptables-alerter-xv89g\" (UID: \"9efa148c-f7c3-4f9d-bee3-cc1eb353b808\") " pod="openshift-network-operator/iptables-alerter-xv89g"
Nov 01 05:44:49.758710 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758522    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-kubelet-config\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:44:49.758710 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758529    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-run-netns\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.758710 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758554    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-kubernetes\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.758710 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758563    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2980ba94-fa03-419a-82d7-86405b6178ac-system-cni-dir\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:49.758710 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758574    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-run-netns\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.758710 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758583    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9efa148c-f7c3-4f9d-bee3-cc1eb353b808-host-slash\") pod \"iptables-alerter-xv89g\" (UID: \"9efa148c-f7c3-4f9d-bee3-cc1eb353b808\") " pod="openshift-network-operator/iptables-alerter-xv89g"
Nov 01 05:44:49.758710 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758590    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:44:49.758710 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758607    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2980ba94-fa03-419a-82d7-86405b6178ac-system-cni-dir\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:49.758710 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758603    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2980ba94-fa03-419a-82d7-86405b6178ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:49.758710 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758662    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2980ba94-fa03-419a-82d7-86405b6178ac-cnibin\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:49.758710 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:49.758698    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 05:44:49.759215 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758747    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-var-lib-kubelet\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.759215 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:49.758764    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret podName:7f767e63-7cc7-4b5e-9941-860fe5f4cd62 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:50.258744764 +0000 UTC m=+2.315347277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret") pod "global-pull-secret-syncer-jh5gk" (UID: "7f767e63-7cc7-4b5e-9941-860fe5f4cd62") : object "kube-system"/"original-pull-secret" not registered
Nov 01 05:44:49.759215 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758788    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-var-lib-kubelet\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.759215 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758794    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86v89\" (UniqueName: \"kubernetes.io/projected/d0e384e0-dcf6-410c-b344-8d797c57e1c9-kube-api-access-86v89\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.759215 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758822    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfpq7\" (UniqueName: \"kubernetes.io/projected/e5ace06a-0f90-46ef-8a04-fdc0f7180a1e-kube-api-access-jfpq7\") pod \"node-ca-dnkvk\" (UID: \"e5ace06a-0f90-46ef-8a04-fdc0f7180a1e\") " pod="openshift-image-registry/node-ca-dnkvk"
Nov 01 05:44:49.759215 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758845    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-var-lib-kubelet\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.759215 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758871    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-run-ovn\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.759215 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758898    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-var-lib-kubelet\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.759215 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758915    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-node-log\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.759215 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758933    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-run-ovn\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.759215 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758943    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-registration-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.759215 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758983    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-etc-selinux\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.759215 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758990    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-registration-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.759215 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.758985    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-node-log\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.759215 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rprhl\" (UniqueName: \"kubernetes.io/projected/9efa148c-f7c3-4f9d-bee3-cc1eb353b808-kube-api-access-rprhl\") pod \"iptables-alerter-xv89g\" (UID: \"9efa148c-f7c3-4f9d-bee3-cc1eb353b808\") " pod="openshift-network-operator/iptables-alerter-xv89g"
Nov 01 05:44:49.759215 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759041    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-sysconfig\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.759876 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759061    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-etc-selinux\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.759876 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759066    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-hostroot\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.759876 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759087    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-sysconfig\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.759876 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759091    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-hostroot\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.759876 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759098    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-kubelet\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.759876 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759103    2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 05:44:49.759876 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759119    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-kubelet\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.759876 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759127    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-log-socket\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.759876 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759146    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-log-socket\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.759876 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759153    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-tuned\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.759876 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759178    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c1d9c47e-944e-4112-9889-b01892353659-env-overrides\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.759876 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759202    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-socket-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.759876 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759227    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e9a8c0a-32d6-44ef-8a47-f65f6560193d-tmp-dir\") pod \"node-resolver-xqcsn\" (UID: \"1e9a8c0a-32d6-44ef-8a47-f65f6560193d\") " pod="openshift-dns/node-resolver-xqcsn"
Nov 01 05:44:49.759876 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759251    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-socket-dir-parent\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.759876 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759276    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-var-lib-cni-bin\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.759876 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759302    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-var-lib-cni-multus\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.759876 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759327    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-dbus\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:44:49.759876 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759372    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdeea076-cd20-459e-b1a5-79ec825e95e3-cni-binary-copy\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.760788 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759413    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-var-lib-cni-bin\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.760788 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759413    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-run-multus-certs\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.760788 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759449    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-device-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.760788 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759456    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-run-multus-certs\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.760788 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759468    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-systemd-units\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.760788 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759500    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-run-systemd\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.760788 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759505    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-var-lib-cni-multus\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.760788 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759530    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-socket-dir-parent\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.760788 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759539    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2980ba94-fa03-419a-82d7-86405b6178ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:49.760788 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759567    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-dbus\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:44:49.760788 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759570    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/05e09ea1-324b-49ef-a4a5-2645b50232a4-konnectivity-ca\") pod \"konnectivity-agent-4lm7r\" (UID: \"05e09ea1-324b-49ef-a4a5-2645b50232a4\") " pod="kube-system/konnectivity-agent-4lm7r"
Nov 01 05:44:49.760788 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759373    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-socket-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.760788 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759599    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c1d9c47e-944e-4112-9889-b01892353659-env-overrides\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.760788 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759606    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-run\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.760788 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759631    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-sys-fs\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.760788 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759642    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-systemd-units\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.760788 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759655    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxjd2\" (UniqueName: \"kubernetes.io/projected/04241194-2fed-4174-9eda-e384dcfd4131-kube-api-access-nxjd2\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.761507 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759676    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-device-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.761507 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759684    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-slash\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.761507 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759708    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-sys-fs\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.761507 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759709    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-etc-openvswitch\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.761507 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759744    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-modprobe-d\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.761507 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759787    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-run-k8s-cni-cncf-io\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.761507 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759805    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e9a8c0a-32d6-44ef-8a47-f65f6560193d-tmp-dir\") pod \"node-resolver-xqcsn\" (UID: \"1e9a8c0a-32d6-44ef-8a47-f65f6560193d\") " pod="openshift-dns/node-resolver-xqcsn"
Nov 01 05:44:49.761507 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759825    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-cni-netd\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.761507 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759857    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-run\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.761507 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759809    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-cni-netd\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.761507 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759897    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-systemd\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.761507 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759919    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-modprobe-d\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.761507 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759924    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-host\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.761507 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759745    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-etc-openvswitch\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.761507 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759951    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-var-lib-openvswitch\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.761507 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759977    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-systemd\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.761507 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759981    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c79lv\" (UniqueName: \"kubernetes.io/projected/c1d9c47e-944e-4112-9889-b01892353659-kube-api-access-c79lv\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.761507 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759976    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-slash\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.762078 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760004    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-run-systemd\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.762078 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760007    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs\") pod \"network-metrics-daemon-gjh6r\" (UID: \"a18b7694-133e-4663-b8f6-a36f981ab717\") " pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:44:49.762078 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760042    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdeea076-cd20-459e-b1a5-79ec825e95e3-cni-binary-copy\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.762078 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760047    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwg57\" (UniqueName: \"kubernetes.io/projected/a18b7694-133e-4663-b8f6-a36f981ab717-kube-api-access-kwg57\") pod \"network-metrics-daemon-gjh6r\" (UID: \"a18b7694-133e-4663-b8f6-a36f981ab717\") " pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:44:49.762078 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760075    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/05e09ea1-324b-49ef-a4a5-2645b50232a4-agent-certs\") pod \"konnectivity-agent-4lm7r\" (UID: \"05e09ea1-324b-49ef-a4a5-2645b50232a4\") " pod="kube-system/konnectivity-agent-4lm7r"
Nov 01 05:44:49.762078 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760083    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-var-lib-openvswitch\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.762078 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760101    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-sysctl-conf\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.762078 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:49.760105    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:44:49.762078 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760118    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/05e09ea1-324b-49ef-a4a5-2645b50232a4-konnectivity-ca\") pod \"konnectivity-agent-4lm7r\" (UID: \"05e09ea1-324b-49ef-a4a5-2645b50232a4\") " pod="kube-system/konnectivity-agent-4lm7r"
Nov 01 05:44:49.762078 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760126    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-lib-modules\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.762078 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.759952    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-run-k8s-cni-cncf-io\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.762078 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760123    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-host\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.762078 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:49.760156    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs podName:a18b7694-133e-4663-b8f6-a36f981ab717 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:50.260138341 +0000 UTC m=+2.316740827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs") pod "network-metrics-daemon-gjh6r" (UID: "a18b7694-133e-4663-b8f6-a36f981ab717") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:44:49.762078 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760180    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58hcx\" (UniqueName: \"kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx\") pod \"network-check-target-cwn9s\" (UID: \"a78d89dc-2ed6-4a2e-9558-f54b73a806d2\") " pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:44:49.762078 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760209    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-kubelet-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.762078 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760237    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c1d9c47e-944e-4112-9889-b01892353659-ovnkube-script-lib\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.762078 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760240    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-lib-modules\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.762699 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760264    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2980ba94-fa03-419a-82d7-86405b6178ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:49.762699 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760294    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5ace06a-0f90-46ef-8a04-fdc0f7180a1e-host\") pod \"node-ca-dnkvk\" (UID: \"e5ace06a-0f90-46ef-8a04-fdc0f7180a1e\") " pod="openshift-image-registry/node-ca-dnkvk"
Nov 01 05:44:49.762699 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760322    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2980ba94-fa03-419a-82d7-86405b6178ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:49.762699 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760370    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-sysctl-d\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.762699 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760418    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0e384e0-dcf6-410c-b344-8d797c57e1c9-tmp\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.762699 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760446    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e9a8c0a-32d6-44ef-8a47-f65f6560193d-hosts-file\") pod \"node-resolver-xqcsn\" (UID: \"1e9a8c0a-32d6-44ef-8a47-f65f6560193d\") " pod="openshift-dns/node-resolver-xqcsn"
Nov 01 05:44:49.762699 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760470    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2980ba94-fa03-419a-82d7-86405b6178ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:49.762699 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-system-cni-dir\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.762699 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760482    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-kubelet-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.762699 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-etc-kubernetes\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.762699 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760531    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.762699 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760612    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.762699 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760776    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-sysctl-d\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.762699 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760876    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-sysctl-conf\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.762699 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760938    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-system-cni-dir\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.762699 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.760993    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e9a8c0a-32d6-44ef-8a47-f65f6560193d-hosts-file\") pod \"node-resolver-xqcsn\" (UID: \"1e9a8c0a-32d6-44ef-8a47-f65f6560193d\") " pod="openshift-dns/node-resolver-xqcsn"
Nov 01 05:44:49.762699 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.761040    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-etc-kubernetes\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.763177 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.761224    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5ace06a-0f90-46ef-8a04-fdc0f7180a1e-host\") pod \"node-ca-dnkvk\" (UID: \"e5ace06a-0f90-46ef-8a04-fdc0f7180a1e\") " pod="openshift-image-registry/node-ca-dnkvk"
Nov 01 05:44:49.763177 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.762476    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c1d9c47e-944e-4112-9889-b01892353659-ovnkube-script-lib\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.763177 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.762573    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2980ba94-fa03-419a-82d7-86405b6178ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:49.763177 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.762620    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2980ba94-fa03-419a-82d7-86405b6178ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:49.763681 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.763652    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0e384e0-dcf6-410c-b344-8d797c57e1c9-tmp\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.763773 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.763739    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1d9c47e-944e-4112-9889-b01892353659-ovn-node-metrics-cert\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.763880 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.763861    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/05e09ea1-324b-49ef-a4a5-2645b50232a4-agent-certs\") pod \"konnectivity-agent-4lm7r\" (UID: \"05e09ea1-324b-49ef-a4a5-2645b50232a4\") " pod="kube-system/konnectivity-agent-4lm7r"
Nov 01 05:44:49.763960 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.763944    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-tuned\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.822790 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.822734    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg2zd\" (UniqueName: \"kubernetes.io/projected/2980ba94-fa03-419a-82d7-86405b6178ac-kube-api-access-vg2zd\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:49.822956 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:49.822939    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 05:44:49.822956 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:49.822958    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 05:44:49.823085 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:49.822968    2583 projected.go:196] Error preparing data for projected volume kube-api-access-58hcx for pod openshift-network-diagnostics/network-check-target-cwn9s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:44:49.823085 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:49.823026    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx podName:a78d89dc-2ed6-4a2e-9558-f54b73a806d2 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:50.323007814 +0000 UTC m=+2.379610321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-58hcx" (UniqueName: "kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx") pod "network-check-target-cwn9s" (UID: "a78d89dc-2ed6-4a2e-9558-f54b73a806d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:44:49.824857 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.824829    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwxk7\" (UniqueName: \"kubernetes.io/projected/bdeea076-cd20-459e-b1a5-79ec825e95e3-kube-api-access-nwxk7\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:49.824949 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.824891    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsndz\" (UniqueName: \"kubernetes.io/projected/1e9a8c0a-32d6-44ef-8a47-f65f6560193d-kube-api-access-tsndz\") pod \"node-resolver-xqcsn\" (UID: \"1e9a8c0a-32d6-44ef-8a47-f65f6560193d\") " pod="openshift-dns/node-resolver-xqcsn"
Nov 01 05:44:49.825173 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.825151    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rprhl\" (UniqueName: \"kubernetes.io/projected/9efa148c-f7c3-4f9d-bee3-cc1eb353b808-kube-api-access-rprhl\") pod \"iptables-alerter-xv89g\" (UID: \"9efa148c-f7c3-4f9d-bee3-cc1eb353b808\") " pod="openshift-network-operator/iptables-alerter-xv89g"
Nov 01 05:44:49.825557 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.825537    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86v89\" (UniqueName: \"kubernetes.io/projected/d0e384e0-dcf6-410c-b344-8d797c57e1c9-kube-api-access-86v89\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:49.825651 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.825571    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfpq7\" (UniqueName: \"kubernetes.io/projected/e5ace06a-0f90-46ef-8a04-fdc0f7180a1e-kube-api-access-jfpq7\") pod \"node-ca-dnkvk\" (UID: \"e5ace06a-0f90-46ef-8a04-fdc0f7180a1e\") " pod="openshift-image-registry/node-ca-dnkvk"
Nov 01 05:44:49.827082 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.827057    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c79lv\" (UniqueName: \"kubernetes.io/projected/c1d9c47e-944e-4112-9889-b01892353659-kube-api-access-c79lv\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.827311 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.827290    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxjd2\" (UniqueName: \"kubernetes.io/projected/04241194-2fed-4174-9eda-e384dcfd4131-kube-api-access-nxjd2\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.827376 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.827314    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwg57\" (UniqueName: \"kubernetes.io/projected/a18b7694-133e-4663-b8f6-a36f981ab717-kube-api-access-kwg57\") pod \"network-metrics-daemon-gjh6r\" (UID: \"a18b7694-133e-4663-b8f6-a36f981ab717\") " pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:44:49.948754 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.948716    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:44:49.961876 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.961839    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xv89g"
Nov 01 05:44:49.978856 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.978830    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4lm7r"
Nov 01 05:44:49.984841 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.984822    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:44:49.992657 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:49.992634    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:44:50.011473 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:50.011456    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:44:50.018148 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:50.018109    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d9c47e_944e_4112_9889_b01892353659.slice/crio-3549a661c8a27497e5587c136c0879c0b79c8774f773f10c552b8e1865c6cbba WatchSource:0}: Error finding container 3549a661c8a27497e5587c136c0879c0b79c8774f773f10c552b8e1865c6cbba: Status 404 returned error can't find the container with id 3549a661c8a27497e5587c136c0879c0b79c8774f773f10c552b8e1865c6cbba
Nov 01 05:44:50.018557 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:50.018526    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e384e0_dcf6_410c_b344_8d797c57e1c9.slice/crio-4695aedc49230336996be4b38c3516576f5102409c521dc26b7743628359b20e WatchSource:0}: Error finding container 4695aedc49230336996be4b38c3516576f5102409c521dc26b7743628359b20e: Status 404 returned error can't find the container with id 4695aedc49230336996be4b38c3516576f5102409c521dc26b7743628359b20e
Nov 01 05:44:50.020294 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:50.019240    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef0f634a7979920e9df606f3bb474aac.slice/crio-9caa216e0fa1aa9115b11851c57c842825c6af055b8bb0e5ec35a0619d90d579 WatchSource:0}: Error finding container 9caa216e0fa1aa9115b11851c57c842825c6af055b8bb0e5ec35a0619d90d579: Status 404 returned error can't find the container with id 9caa216e0fa1aa9115b11851c57c842825c6af055b8bb0e5ec35a0619d90d579
Nov 01 05:44:50.020294 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:50.019983    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e09ea1_324b_49ef_a4a5_2645b50232a4.slice/crio-4f7a80cdd4eb9a036dcb5c7db83b97cf9a20d11260749a6dc93fe3d1e61f12f4 WatchSource:0}: Error finding container 4f7a80cdd4eb9a036dcb5c7db83b97cf9a20d11260749a6dc93fe3d1e61f12f4: Status 404 returned error can't find the container with id 4f7a80cdd4eb9a036dcb5c7db83b97cf9a20d11260749a6dc93fe3d1e61f12f4
Nov 01 05:44:50.024023 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:50.023994    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 05:44:50.027012 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:50.026982    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a06cb4c587db3aac34f3d8afd3bbd7.slice/crio-6a64fa9dac836f74899877d2701dccd8af6f843a2ca1367131f63ce41196841a WatchSource:0}: Error finding container 6a64fa9dac836f74899877d2701dccd8af6f843a2ca1367131f63ce41196841a: Status 404 returned error can't find the container with id 6a64fa9dac836f74899877d2701dccd8af6f843a2ca1367131f63ce41196841a
Nov 01 05:44:50.028386 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:50.028280    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2980ba94_fa03_419a_82d7_86405b6178ac.slice/crio-41eba9dd5e22a6c3b1e9cb7102d76937242618bd0230e92edda9fa423e5a6622 WatchSource:0}: Error finding container 41eba9dd5e22a6c3b1e9cb7102d76937242618bd0230e92edda9fa423e5a6622: Status 404 returned error can't find the container with id 41eba9dd5e22a6c3b1e9cb7102d76937242618bd0230e92edda9fa423e5a6622
Nov 01 05:44:50.031584 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:50.031564    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xqcsn"
Nov 01 05:44:50.040584 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:50.040564    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9a8c0a_32d6_44ef_8a47_f65f6560193d.slice/crio-15707454db3c61e3767aa29b3d5d424b86a1a1094ac8904dc79e0760c4f332bb WatchSource:0}: Error finding container 15707454db3c61e3767aa29b3d5d424b86a1a1094ac8904dc79e0760c4f332bb: Status 404 returned error can't find the container with id 15707454db3c61e3767aa29b3d5d424b86a1a1094ac8904dc79e0760c4f332bb
Nov 01 05:44:50.046708 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:50.046690    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dnkvk"
Nov 01 05:44:50.051685 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:50.051671    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7xz4d"
Nov 01 05:44:50.056292 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:50.056272    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ace06a_0f90_46ef_8a04_fdc0f7180a1e.slice/crio-4bab4c0a43a5777b39b5767479c6853f715e7c24e8353ac3e500686a43aef09f WatchSource:0}: Error finding container 4bab4c0a43a5777b39b5767479c6853f715e7c24e8353ac3e500686a43aef09f: Status 404 returned error can't find the container with id 4bab4c0a43a5777b39b5767479c6853f715e7c24e8353ac3e500686a43aef09f
Nov 01 05:44:50.060085 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:50.060067    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdeea076_cd20_459e_b1a5_79ec825e95e3.slice/crio-b0330c552dede6cbfd8eca1d6ca5c65eaa9a470af6a0338c93d30f2cf853f350 WatchSource:0}: Error finding container b0330c552dede6cbfd8eca1d6ca5c65eaa9a470af6a0338c93d30f2cf853f350: Status 404 returned error can't find the container with id b0330c552dede6cbfd8eca1d6ca5c65eaa9a470af6a0338c93d30f2cf853f350
Nov 01 05:44:50.165368 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:50.165305    2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:44:50.263922 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:50.263890    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:44:50.264060 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:50.263940    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs\") pod \"network-metrics-daemon-gjh6r\" (UID: \"a18b7694-133e-4663-b8f6-a36f981ab717\") " pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:44:50.264060 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:50.264044    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:44:50.264060 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:50.264045    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 05:44:50.264158 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:50.264098    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs podName:a18b7694-133e-4663-b8f6-a36f981ab717 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:51.26408095 +0000 UTC m=+3.320683453 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs") pod "network-metrics-daemon-gjh6r" (UID: "a18b7694-133e-4663-b8f6-a36f981ab717") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:44:50.264158 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:50.264111    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret podName:7f767e63-7cc7-4b5e-9941-860fe5f4cd62 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:51.264105382 +0000 UTC m=+3.320707865 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret") pod "global-pull-secret-syncer-jh5gk" (UID: "7f767e63-7cc7-4b5e-9941-860fe5f4cd62") : object "kube-system"/"original-pull-secret" not registered
Nov 01 05:44:50.364872 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:50.364840    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58hcx\" (UniqueName: \"kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx\") pod \"network-check-target-cwn9s\" (UID: \"a78d89dc-2ed6-4a2e-9558-f54b73a806d2\") " pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:44:50.365348 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:50.364976    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 05:44:50.365348 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:50.364990    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 05:44:50.365348 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:50.364999    2583 projected.go:196] Error preparing data for projected volume kube-api-access-58hcx for pod openshift-network-diagnostics/network-check-target-cwn9s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:44:50.365348 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:50.365049    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx podName:a78d89dc-2ed6-4a2e-9558-f54b73a806d2 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:51.365036193 +0000 UTC m=+3.421638675 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-58hcx" (UniqueName: "kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx") pod "network-check-target-cwn9s" (UID: "a78d89dc-2ed6-4a2e-9558-f54b73a806d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:44:50.468397 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:50.468294    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7xz4d" event={"ID":"bdeea076-cd20-459e-b1a5-79ec825e95e3","Type":"ContainerStarted","Data":"b0330c552dede6cbfd8eca1d6ca5c65eaa9a470af6a0338c93d30f2cf853f350"}
Nov 01 05:44:50.470932 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:50.470423    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dnkvk" event={"ID":"e5ace06a-0f90-46ef-8a04-fdc0f7180a1e","Type":"ContainerStarted","Data":"4bab4c0a43a5777b39b5767479c6853f715e7c24e8353ac3e500686a43aef09f"}
Nov 01 05:44:50.472421 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:50.472384    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xqcsn" event={"ID":"1e9a8c0a-32d6-44ef-8a47-f65f6560193d","Type":"ContainerStarted","Data":"15707454db3c61e3767aa29b3d5d424b86a1a1094ac8904dc79e0760c4f332bb"}
Nov 01 05:44:50.473556 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:50.473534    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal" event={"ID":"28a06cb4c587db3aac34f3d8afd3bbd7","Type":"ContainerStarted","Data":"6a64fa9dac836f74899877d2701dccd8af6f843a2ca1367131f63ce41196841a"}
Nov 01 05:44:50.474616 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:50.474593    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z29gq" event={"ID":"d0e384e0-dcf6-410c-b344-8d797c57e1c9","Type":"ContainerStarted","Data":"4695aedc49230336996be4b38c3516576f5102409c521dc26b7743628359b20e"}
Nov 01 05:44:50.476244 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:50.476223    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4lm7r" event={"ID":"05e09ea1-324b-49ef-a4a5-2645b50232a4","Type":"ContainerStarted","Data":"4f7a80cdd4eb9a036dcb5c7db83b97cf9a20d11260749a6dc93fe3d1e61f12f4"}
Nov 01 05:44:50.477311 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:50.477286    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4ps2" event={"ID":"2980ba94-fa03-419a-82d7-86405b6178ac","Type":"ContainerStarted","Data":"41eba9dd5e22a6c3b1e9cb7102d76937242618bd0230e92edda9fa423e5a6622"}
Nov 01 05:44:50.478386 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:50.478367    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal" event={"ID":"ef0f634a7979920e9df606f3bb474aac","Type":"ContainerStarted","Data":"9caa216e0fa1aa9115b11851c57c842825c6af055b8bb0e5ec35a0619d90d579"}
Nov 01 05:44:50.479514 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:50.479496    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nt68j" event={"ID":"c1d9c47e-944e-4112-9889-b01892353659","Type":"ContainerStarted","Data":"3549a661c8a27497e5587c136c0879c0b79c8774f773f10c552b8e1865c6cbba"}
Nov 01 05:44:50.636681 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:50.636650    2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:44:50.665352 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:50.665315    2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:44:50.735551 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:50.735475    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 05:39:49 +0000 UTC" deadline="2028-06-24 22:46:04.121100676 +0000 UTC"
Nov 01 05:44:50.735551 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:50.735506    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14441h1m13.385597712s"
Nov 01 05:44:50.908572 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:50.908542    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9efa148c_f7c3_4f9d_bee3_cc1eb353b808.slice/crio-35a484ca0d3ce1bf1d8f66d1bba9a8a4220a5664e20143504c690bb7277f7642 WatchSource:0}: Error finding container 35a484ca0d3ce1bf1d8f66d1bba9a8a4220a5664e20143504c690bb7277f7642: Status 404 returned error can't find the container with id 35a484ca0d3ce1bf1d8f66d1bba9a8a4220a5664e20143504c690bb7277f7642
Nov 01 05:44:50.909296 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:44:50.909280    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04241194_2fed_4174_9eda_e384dcfd4131.slice/crio-af3164d5e42d1871cc9e2571d7ed0458fc9ce0c6b2f407cb740e58d571f0d652 WatchSource:0}: Error finding container af3164d5e42d1871cc9e2571d7ed0458fc9ce0c6b2f407cb740e58d571f0d652: Status 404 returned error can't find the container with id af3164d5e42d1871cc9e2571d7ed0458fc9ce0c6b2f407cb740e58d571f0d652
Nov 01 05:44:51.274396 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:51.274335    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs\") pod \"network-metrics-daemon-gjh6r\" (UID: \"a18b7694-133e-4663-b8f6-a36f981ab717\") " pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:44:51.274594 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:51.274427    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:44:51.274594 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:51.274555    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 05:44:51.274770 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:51.274619    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret podName:7f767e63-7cc7-4b5e-9941-860fe5f4cd62 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:53.274601024 +0000 UTC m=+5.331203520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret") pod "global-pull-secret-syncer-jh5gk" (UID: "7f767e63-7cc7-4b5e-9941-860fe5f4cd62") : object "kube-system"/"original-pull-secret" not registered
Nov 01 05:44:51.274770 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:51.274693    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:44:51.274770 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:51.274731    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs podName:a18b7694-133e-4663-b8f6-a36f981ab717 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:53.274718096 +0000 UTC m=+5.331320593 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs") pod "network-metrics-daemon-gjh6r" (UID: "a18b7694-133e-4663-b8f6-a36f981ab717") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:44:51.375833 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:51.375791    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58hcx\" (UniqueName: \"kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx\") pod \"network-check-target-cwn9s\" (UID: \"a78d89dc-2ed6-4a2e-9558-f54b73a806d2\") " pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:44:51.376343 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:51.375971    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 05:44:51.376343 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:51.375997    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 05:44:51.376343 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:51.376012    2583 projected.go:196] Error preparing data for projected volume kube-api-access-58hcx for pod openshift-network-diagnostics/network-check-target-cwn9s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:44:51.376343 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:51.376075    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx podName:a78d89dc-2ed6-4a2e-9558-f54b73a806d2 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:53.376056191 +0000 UTC m=+5.432658691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-58hcx" (UniqueName: "kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx") pod "network-check-target-cwn9s" (UID: "a78d89dc-2ed6-4a2e-9558-f54b73a806d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:44:51.465020 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:51.464981    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:44:51.465197 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:51.465134    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cwn9s" podUID="a78d89dc-2ed6-4a2e-9558-f54b73a806d2"
Nov 01 05:44:51.465272 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:51.465229    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:44:51.465334 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:51.465319    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjh6r" podUID="a18b7694-133e-4663-b8f6-a36f981ab717"
Nov 01 05:44:51.465443 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:51.465405    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:44:51.465525 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:51.465488    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jh5gk" podUID="7f767e63-7cc7-4b5e-9941-860fe5f4cd62"
Nov 01 05:44:51.494790 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:51.494756    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s" event={"ID":"04241194-2fed-4174-9eda-e384dcfd4131","Type":"ContainerStarted","Data":"af3164d5e42d1871cc9e2571d7ed0458fc9ce0c6b2f407cb740e58d571f0d652"}
Nov 01 05:44:51.506088 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:51.506041    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xv89g" event={"ID":"9efa148c-f7c3-4f9d-bee3-cc1eb353b808","Type":"ContainerStarted","Data":"35a484ca0d3ce1bf1d8f66d1bba9a8a4220a5664e20143504c690bb7277f7642"}
Nov 01 05:44:51.736350 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:51.736220    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 05:39:49 +0000 UTC" deadline="2028-07-05 20:24:09.963718753 +0000 UTC"
Nov 01 05:44:51.736350 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:51.736283    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14702h39m18.227439694s"
Nov 01 05:44:53.293218 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:53.292674    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs\") pod \"network-metrics-daemon-gjh6r\" (UID: \"a18b7694-133e-4663-b8f6-a36f981ab717\") " pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:44:53.293218 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:53.292760    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:44:53.293218 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:53.292895    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 05:44:53.293218 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:53.292960    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret podName:7f767e63-7cc7-4b5e-9941-860fe5f4cd62 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:57.292942112 +0000 UTC m=+9.349544612 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret") pod "global-pull-secret-syncer-jh5gk" (UID: "7f767e63-7cc7-4b5e-9941-860fe5f4cd62") : object "kube-system"/"original-pull-secret" not registered
Nov 01 05:44:53.293218 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:53.293031    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:44:53.293218 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:53.293070    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs podName:a18b7694-133e-4663-b8f6-a36f981ab717 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:57.293056245 +0000 UTC m=+9.349658730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs") pod "network-metrics-daemon-gjh6r" (UID: "a18b7694-133e-4663-b8f6-a36f981ab717") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:44:53.393777 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:53.393743    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58hcx\" (UniqueName: \"kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx\") pod \"network-check-target-cwn9s\" (UID: \"a78d89dc-2ed6-4a2e-9558-f54b73a806d2\") " pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:44:53.394269 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:53.393949    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 05:44:53.394269 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:53.393973    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 05:44:53.394269 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:53.393987    2583 projected.go:196] Error preparing data for projected volume kube-api-access-58hcx for pod openshift-network-diagnostics/network-check-target-cwn9s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:44:53.394269 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:53.394048    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx podName:a78d89dc-2ed6-4a2e-9558-f54b73a806d2 nodeName:}" failed. No retries permitted until 2026-11-01 05:44:57.394029123 +0000 UTC m=+9.450631644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-58hcx" (UniqueName: "kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx") pod "network-check-target-cwn9s" (UID: "a78d89dc-2ed6-4a2e-9558-f54b73a806d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:44:53.465745 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:53.465224    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:44:53.465745 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:53.465380    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jh5gk" podUID="7f767e63-7cc7-4b5e-9941-860fe5f4cd62"
Nov 01 05:44:53.465745 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:53.465457    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:44:53.465745 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:53.465538    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cwn9s" podUID="a78d89dc-2ed6-4a2e-9558-f54b73a806d2"
Nov 01 05:44:53.465745 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:53.465595    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:44:53.465745 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:53.465695    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjh6r" podUID="a18b7694-133e-4663-b8f6-a36f981ab717"
Nov 01 05:44:54.532510 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:54.531146    2583 generic.go:356] "Generic (PLEG): container finished" podID="2980ba94-fa03-419a-82d7-86405b6178ac" containerID="5b61f46fa443e030a233667f724107f0a964e69e71ef468642e66501d8b46bdf" exitCode=0
Nov 01 05:44:54.532510 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:54.531210    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4ps2" event={"ID":"2980ba94-fa03-419a-82d7-86405b6178ac","Type":"ContainerDied","Data":"5b61f46fa443e030a233667f724107f0a964e69e71ef468642e66501d8b46bdf"}
Nov 01 05:44:55.465158 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:55.464785    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:44:55.465158 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:55.465035    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:44:55.465371 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:55.465166    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjh6r" podUID="a18b7694-133e-4663-b8f6-a36f981ab717"
Nov 01 05:44:55.465371 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:55.464988    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:44:55.465371 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:55.465275    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jh5gk" podUID="7f767e63-7cc7-4b5e-9941-860fe5f4cd62"
Nov 01 05:44:55.465371 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:55.465354    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cwn9s" podUID="a78d89dc-2ed6-4a2e-9558-f54b73a806d2"
Nov 01 05:44:55.542804 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:55.542565    2583 generic.go:356] "Generic (PLEG): container finished" podID="28a06cb4c587db3aac34f3d8afd3bbd7" containerID="41139941351a4601d5171e7c8eff0a7feddffde463f7d759201ef7a8f4ec8ce9" exitCode=0
Nov 01 05:44:55.542804 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:55.542715    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal" event={"ID":"28a06cb4c587db3aac34f3d8afd3bbd7","Type":"ContainerDied","Data":"41139941351a4601d5171e7c8eff0a7feddffde463f7d759201ef7a8f4ec8ce9"}
Nov 01 05:44:55.547682 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:55.547650    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4lm7r" event={"ID":"05e09ea1-324b-49ef-a4a5-2645b50232a4","Type":"ContainerStarted","Data":"612d56022baaaf26730a479afd8399b30618c6c1873f41561c40e2fb3116c70c"}
Nov 01 05:44:56.342589 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:56.342557    2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4lm7r"
Nov 01 05:44:56.343119 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:56.343099    2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4lm7r"
Nov 01 05:44:56.457269 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:56.457207    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4lm7r" podStartSLOduration=2.5226533 podStartE2EDuration="7.457186871s" totalImagesPullingTime="4.934533571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:55.793695243 +0000 UTC m=+7.850297762" watchObservedRunningTime="2026-11-01 05:44:56.457186871 +0000 UTC m=+8.513789375"
Nov 01 05:44:56.558380 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:56.557629    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal" event={"ID":"ef0f634a7979920e9df606f3bb474aac","Type":"ContainerStarted","Data":"e06d6d1f41d4df680996c5c7adf8706e59331aeb69d06d0be2bfdd68af2b98cb"}
Nov 01 05:44:56.563673 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:56.563394    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal" event={"ID":"28a06cb4c587db3aac34f3d8afd3bbd7","Type":"ContainerStarted","Data":"109981c288c5c2a2a54b1f820e095bf3c7b1274d1ce9ef2bdd8665903bf1a859"}
Nov 01 05:44:56.670723 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:56.670088    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal" podStartSLOduration=7.670073471 podStartE2EDuration="7.670073471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:56.669553543 +0000 UTC m=+8.726156047" watchObservedRunningTime="2026-11-01 05:44:56.670073471 +0000 UTC m=+8.726675970"
Nov 01 05:44:57.327636 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:57.327557    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs\") pod \"network-metrics-daemon-gjh6r\" (UID: \"a18b7694-133e-4663-b8f6-a36f981ab717\") " pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:44:57.327794 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:57.327648    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:44:57.327794 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:57.327782    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 05:44:57.327897 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:57.327833    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret podName:7f767e63-7cc7-4b5e-9941-860fe5f4cd62 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:05.327818898 +0000 UTC m=+17.384421392 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret") pod "global-pull-secret-syncer-jh5gk" (UID: "7f767e63-7cc7-4b5e-9941-860fe5f4cd62") : object "kube-system"/"original-pull-secret" not registered
Nov 01 05:44:57.327897 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:57.327883    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:44:57.327964 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:57.327905    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs podName:a18b7694-133e-4663-b8f6-a36f981ab717 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:05.327898687 +0000 UTC m=+17.384501167 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs") pod "network-metrics-daemon-gjh6r" (UID: "a18b7694-133e-4663-b8f6-a36f981ab717") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:44:57.428115 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:57.428076    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58hcx\" (UniqueName: \"kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx\") pod \"network-check-target-cwn9s\" (UID: \"a78d89dc-2ed6-4a2e-9558-f54b73a806d2\") " pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:44:57.428476 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:57.428313    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 05:44:57.428548 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:57.428495    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 05:44:57.428548 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:57.428511    2583 projected.go:196] Error preparing data for projected volume kube-api-access-58hcx for pod openshift-network-diagnostics/network-check-target-cwn9s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:44:57.428656 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:57.428588    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx podName:a78d89dc-2ed6-4a2e-9558-f54b73a806d2 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:05.428567288 +0000 UTC m=+17.485169791 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-58hcx" (UniqueName: "kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx") pod "network-check-target-cwn9s" (UID: "a78d89dc-2ed6-4a2e-9558-f54b73a806d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:44:57.465314 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:57.465287    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:44:57.465483 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:57.465398    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cwn9s" podUID="a78d89dc-2ed6-4a2e-9558-f54b73a806d2"
Nov 01 05:44:57.465483 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:57.465467    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:44:57.465581 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:57.465524    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjh6r" podUID="a18b7694-133e-4663-b8f6-a36f981ab717"
Nov 01 05:44:57.465644 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:57.465583    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:44:57.465644 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:57.465627    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jh5gk" podUID="7f767e63-7cc7-4b5e-9941-860fe5f4cd62"
Nov 01 05:44:57.568445 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:57.568413    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dnkvk" event={"ID":"e5ace06a-0f90-46ef-8a04-fdc0f7180a1e","Type":"ContainerStarted","Data":"59971ff932ab671cb1cd4696d0193547f349f98ff449c1cd02e3adbda25d70e6"}
Nov 01 05:44:57.568908 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:57.568507    2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 05:44:57.681217 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:57.681157    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dnkvk" podStartSLOduration=2.328632298 podStartE2EDuration="8.681140259s" totalImagesPullingTime="6.352507961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:57.680433654 +0000 UTC m=+9.737036158" watchObservedRunningTime="2026-11-01 05:44:57.681140259 +0000 UTC m=+9.737742761"
Nov 01 05:44:57.681615 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:57.681580    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal" podStartSLOduration=8.681567721 podStartE2EDuration="8.681567721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:56.785251002 +0000 UTC m=+8.841853505" watchObservedRunningTime="2026-11-01 05:44:57.681567721 +0000 UTC m=+9.738170244"
Nov 01 05:44:58.571536 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:58.571501    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s" event={"ID":"04241194-2fed-4174-9eda-e384dcfd4131","Type":"ContainerStarted","Data":"c3a516046a51d46b44c5fbfe42bbd5ea3a54c5282d393e02487335ff7a5335e4"}
Nov 01 05:44:59.464799 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:59.464686    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:44:59.464934 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:59.464802    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cwn9s" podUID="a78d89dc-2ed6-4a2e-9558-f54b73a806d2"
Nov 01 05:44:59.464934 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:59.464865    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:44:59.465089 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:59.464933    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjh6r" podUID="a18b7694-133e-4663-b8f6-a36f981ab717"
Nov 01 05:44:59.465089 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:59.464979    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:44:59.465089 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:44:59.465032    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jh5gk" podUID="7f767e63-7cc7-4b5e-9941-860fe5f4cd62"
Nov 01 05:44:59.578399 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:59.577780    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xqcsn" event={"ID":"1e9a8c0a-32d6-44ef-8a47-f65f6560193d","Type":"ContainerStarted","Data":"86cb996ca9ec0e11c600db64665e9f3512d913ac7c807db6ed6e082c885e5d6f"}
Nov 01 05:44:59.584707 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:59.582651    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7xz4d" event={"ID":"bdeea076-cd20-459e-b1a5-79ec825e95e3","Type":"ContainerStarted","Data":"61161559d556b3e13b5b4a5b9687b3b1af5fda5a55942bb434960971a81fcba7"}
Nov 01 05:44:59.629415 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:59.629370    2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 05:44:59.692542 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:59.692489    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-xqcsn" podStartSLOduration=1.638393795 podStartE2EDuration="10.692473123s" totalImagesPullingTime="9.054079328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:59.69192276 +0000 UTC m=+11.748525265" watchObservedRunningTime="2026-11-01 05:44:59.692473123 +0000 UTC m=+11.749075625"
Nov 01 05:44:59.808415 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:44:59.808364    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-7xz4d" podStartSLOduration=2.103109022 podStartE2EDuration="10.808348515s" totalImagesPullingTime="8.705239493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:44:59.807774674 +0000 UTC m=+11.864377175" watchObservedRunningTime="2026-11-01 05:44:59.808348515 +0000 UTC m=+11.864951017"
Nov 01 05:45:00.421076 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:00.420941    2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T05:44:59.629408045Z","UUID":"5cf7c606-fd35-4c74-a3fb-5fd887bab878","Handler":null,"Name":"","Endpoint":""}
Nov 01 05:45:00.422482 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:00.422463    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 05:45:00.422482 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:00.422490    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 05:45:00.587739 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:00.587661    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s" event={"ID":"04241194-2fed-4174-9eda-e384dcfd4131","Type":"ContainerStarted","Data":"91a1c3d3c9500adf5e3e7360a78755f60e1c5d118c2196957a7c2afb19fa5d65"}
Nov 01 05:45:00.590315 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:00.590286    2583 generic.go:356] "Generic (PLEG): container finished" podID="2980ba94-fa03-419a-82d7-86405b6178ac" containerID="c25f4e50ad87fa964adc6b9538fecde359cf135a451acb50c9d6be1ffddd4078" exitCode=0
Nov 01 05:45:00.590475 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:00.590353    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4ps2" event={"ID":"2980ba94-fa03-419a-82d7-86405b6178ac","Type":"ContainerDied","Data":"c25f4e50ad87fa964adc6b9538fecde359cf135a451acb50c9d6be1ffddd4078"}
Nov 01 05:45:01.465044 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:01.465009    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:45:01.465195 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:01.465084    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:45:01.465195 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:01.465174    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:45:01.465195 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:01.465179    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jh5gk" podUID="7f767e63-7cc7-4b5e-9941-860fe5f4cd62"
Nov 01 05:45:01.465288 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:01.465246    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjh6r" podUID="a18b7694-133e-4663-b8f6-a36f981ab717"
Nov 01 05:45:01.465352 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:01.465319    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cwn9s" podUID="a78d89dc-2ed6-4a2e-9558-f54b73a806d2"
Nov 01 05:45:01.593836 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:01.593802    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xv89g" event={"ID":"9efa148c-f7c3-4f9d-bee3-cc1eb353b808","Type":"ContainerStarted","Data":"8738b3a993b45a1286154e6f9ca7fd12aa81970ac416ac8a8193f466605ee2e0"}
Nov 01 05:45:01.708721 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:01.708670    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xv89g" podStartSLOduration=4.067322982 podStartE2EDuration="12.708657555s" totalImagesPullingTime="8.641334573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:01.708371183 +0000 UTC m=+13.764973685" watchObservedRunningTime="2026-11-01 05:45:01.708657555 +0000 UTC m=+13.765260056"
Nov 01 05:45:02.597926 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:02.597700    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z29gq" event={"ID":"d0e384e0-dcf6-410c-b344-8d797c57e1c9","Type":"ContainerStarted","Data":"9a099f24f7c9f56c56094f6a0dc505afead6d139f30769d8160fd286a80a4ddd"}
Nov 01 05:45:02.599903 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:02.599858    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s" event={"ID":"04241194-2fed-4174-9eda-e384dcfd4131","Type":"ContainerStarted","Data":"0a700c19f5f0d7db1de495ef0a2f334de107f3be82f5b0604265543354bd2762"}
Nov 01 05:45:02.601825 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:02.601797    2583 generic.go:356] "Generic (PLEG): container finished" podID="2980ba94-fa03-419a-82d7-86405b6178ac" containerID="d905f4c1a2c65acd1023f316e58c750290a7ceb3e9dc4148c1211c9beb8cd1d7" exitCode=0
Nov 01 05:45:02.601922 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:02.601879    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4ps2" event={"ID":"2980ba94-fa03-419a-82d7-86405b6178ac","Type":"ContainerDied","Data":"d905f4c1a2c65acd1023f316e58c750290a7ceb3e9dc4148c1211c9beb8cd1d7"}
Nov 01 05:45:02.713853 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:02.713801    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-z29gq" podStartSLOduration=2.025211237 podStartE2EDuration="13.713786252s" totalImagesPullingTime="11.688575015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:02.713453339 +0000 UTC m=+14.770055842" watchObservedRunningTime="2026-11-01 05:45:02.713786252 +0000 UTC m=+14.770388753"
Nov 01 05:45:02.953689 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:02.953601    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s" podStartSLOduration=3.472418089 podStartE2EDuration="13.95358384s" totalImagesPullingTime="10.481165751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:02.953507862 +0000 UTC m=+15.010110364" watchObservedRunningTime="2026-11-01 05:45:02.95358384 +0000 UTC m=+15.010186342"
Nov 01 05:45:03.464465 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:03.464433    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:45:03.464621 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:03.464520    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:45:03.464621 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:03.464542    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:45:03.464740 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:03.464643    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jh5gk" podUID="7f767e63-7cc7-4b5e-9941-860fe5f4cd62"
Nov 01 05:45:03.464788 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:03.464756    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjh6r" podUID="a18b7694-133e-4663-b8f6-a36f981ab717"
Nov 01 05:45:03.464846 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:03.464828    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cwn9s" podUID="a78d89dc-2ed6-4a2e-9558-f54b73a806d2"
Nov 01 05:45:04.608457 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:04.608422    2583 generic.go:356] "Generic (PLEG): container finished" podID="2980ba94-fa03-419a-82d7-86405b6178ac" containerID="ae1ddcd938354d5d2782f4e1bd7c14a2ec400c49929176e0b34bae2287359e05" exitCode=0
Nov 01 05:45:04.609173 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:04.608475    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4ps2" event={"ID":"2980ba94-fa03-419a-82d7-86405b6178ac","Type":"ContainerDied","Data":"ae1ddcd938354d5d2782f4e1bd7c14a2ec400c49929176e0b34bae2287359e05"}
Nov 01 05:45:05.393581 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:05.393549    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:45:05.393768 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:05.393597    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs\") pod \"network-metrics-daemon-gjh6r\" (UID: \"a18b7694-133e-4663-b8f6-a36f981ab717\") " pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:45:05.393768 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:05.393714    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:45:05.393891 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:05.393774    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs podName:a18b7694-133e-4663-b8f6-a36f981ab717 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:21.393761659 +0000 UTC m=+33.450364152 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs") pod "network-metrics-daemon-gjh6r" (UID: "a18b7694-133e-4663-b8f6-a36f981ab717") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:45:05.393891 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:05.393712    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 05:45:05.393891 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:05.393868    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret podName:7f767e63-7cc7-4b5e-9941-860fe5f4cd62 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:21.393848785 +0000 UTC m=+33.450451270 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret") pod "global-pull-secret-syncer-jh5gk" (UID: "7f767e63-7cc7-4b5e-9941-860fe5f4cd62") : object "kube-system"/"original-pull-secret" not registered
Nov 01 05:45:05.464902 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:05.464863    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:45:05.465058 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:05.464910    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:45:05.465058 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:05.464877    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:45:05.465058 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:05.465010    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjh6r" podUID="a18b7694-133e-4663-b8f6-a36f981ab717"
Nov 01 05:45:05.465178 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:05.465078    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jh5gk" podUID="7f767e63-7cc7-4b5e-9941-860fe5f4cd62"
Nov 01 05:45:05.465178 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:05.465155    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cwn9s" podUID="a78d89dc-2ed6-4a2e-9558-f54b73a806d2"
Nov 01 05:45:05.494588 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:05.494559    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58hcx\" (UniqueName: \"kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx\") pod \"network-check-target-cwn9s\" (UID: \"a78d89dc-2ed6-4a2e-9558-f54b73a806d2\") " pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:45:05.494781 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:05.494751    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 05:45:05.494781 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:05.494783    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 05:45:05.494957 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:05.494794    2583 projected.go:196] Error preparing data for projected volume kube-api-access-58hcx for pod openshift-network-diagnostics/network-check-target-cwn9s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:45:05.494957 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:05.494854    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx podName:a78d89dc-2ed6-4a2e-9558-f54b73a806d2 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:21.494837059 +0000 UTC m=+33.551439542 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-58hcx" (UniqueName: "kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx") pod "network-check-target-cwn9s" (UID: "a78d89dc-2ed6-4a2e-9558-f54b73a806d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:45:07.464847 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:07.464807    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:45:07.465339 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:07.464877    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:45:07.465339 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:07.464991    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:45:07.465339 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:07.465008    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cwn9s" podUID="a78d89dc-2ed6-4a2e-9558-f54b73a806d2"
Nov 01 05:45:07.465339 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:07.465108    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjh6r" podUID="a18b7694-133e-4663-b8f6-a36f981ab717"
Nov 01 05:45:07.465339 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:07.465183    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jh5gk" podUID="7f767e63-7cc7-4b5e-9941-860fe5f4cd62"
Nov 01 05:45:07.618250 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:07.618214    2583 generic.go:356] "Generic (PLEG): container finished" podID="2980ba94-fa03-419a-82d7-86405b6178ac" containerID="6f5a7e9fd77d3a850d78b77c143caf23e8653bc60d4fdda10d1ee5df08d5b6c2" exitCode=0
Nov 01 05:45:07.618425 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:07.618272    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4ps2" event={"ID":"2980ba94-fa03-419a-82d7-86405b6178ac","Type":"ContainerDied","Data":"6f5a7e9fd77d3a850d78b77c143caf23e8653bc60d4fdda10d1ee5df08d5b6c2"}
Nov 01 05:45:08.626478 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:08.626179    2583 generic.go:356] "Generic (PLEG): container finished" podID="2980ba94-fa03-419a-82d7-86405b6178ac" containerID="541e8cdc298f1f321484d562988b6dbac91f943703c3f2b2eea3fcec3d0044b3" exitCode=0
Nov 01 05:45:08.626950 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:08.626254    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4ps2" event={"ID":"2980ba94-fa03-419a-82d7-86405b6178ac","Type":"ContainerDied","Data":"541e8cdc298f1f321484d562988b6dbac91f943703c3f2b2eea3fcec3d0044b3"}
Nov 01 05:45:08.629572 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:08.629550    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nt68j" event={"ID":"c1d9c47e-944e-4112-9889-b01892353659","Type":"ContainerStarted","Data":"f553df07f6a30addd7fc34605d31b01695b279c76fc03c407ec588616900420a"}
Nov 01 05:45:08.629671 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:08.629602    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nt68j" event={"ID":"c1d9c47e-944e-4112-9889-b01892353659","Type":"ContainerStarted","Data":"ecb9753488d4ea64f6f9e0902b479f6f89b7ff23f9a8f8d541a9e22555e23767"}
Nov 01 05:45:08.629671 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:08.629632    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nt68j" event={"ID":"c1d9c47e-944e-4112-9889-b01892353659","Type":"ContainerStarted","Data":"3222e3fc52948efbc2467d27a422d6dc224276dfb8e08a7a1d4a277d1dabc033"}
Nov 01 05:45:08.629671 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:08.629659    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nt68j" event={"ID":"c1d9c47e-944e-4112-9889-b01892353659","Type":"ContainerStarted","Data":"c448f8b7c1be8aceb454e607fb853eceae6996ded916dfa1a87d892b55298292"}
Nov 01 05:45:08.629834 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:08.629687    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nt68j" event={"ID":"c1d9c47e-944e-4112-9889-b01892353659","Type":"ContainerStarted","Data":"af73e1fc437e115e6633dbd20280318b927cfb362c89830ef6561dbdf72ee639"}
Nov 01 05:45:08.629834 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:08.629713    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nt68j" event={"ID":"c1d9c47e-944e-4112-9889-b01892353659","Type":"ContainerStarted","Data":"2b35515567366279c75f9041a9baffd7d20c4aa8c709aba85f216eb8058f8dfd"}
Nov 01 05:45:09.465076 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:09.464914    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:45:09.465219 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:09.464980    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:45:09.465219 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:09.465145    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cwn9s" podUID="a78d89dc-2ed6-4a2e-9558-f54b73a806d2"
Nov 01 05:45:09.465219 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:09.464987    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:45:09.465219 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:09.465213    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jh5gk" podUID="7f767e63-7cc7-4b5e-9941-860fe5f4cd62"
Nov 01 05:45:09.465399 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:09.465300    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjh6r" podUID="a18b7694-133e-4663-b8f6-a36f981ab717"
Nov 01 05:45:09.634711 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:09.634678    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4ps2" event={"ID":"2980ba94-fa03-419a-82d7-86405b6178ac","Type":"ContainerStarted","Data":"f3615f9f8a4040b797435a362f569e07064961daa95c09256c2a044e28576959"}
Nov 01 05:45:09.754970 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:09.754883    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h4ps2" podStartSLOduration=7.476914643 podStartE2EDuration="20.754868924s" totalImagesPullingTime="13.277954281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:09.754624089 +0000 UTC m=+21.811226592" watchObservedRunningTime="2026-11-01 05:45:09.754868924 +0000 UTC m=+21.811471426"
Nov 01 05:45:09.877423 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:09.877387    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:10.098562 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:10.098483    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:10.285500 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:10.285465    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:10.639469 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:10.639373    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nt68j" event={"ID":"c1d9c47e-944e-4112-9889-b01892353659","Type":"ContainerStarted","Data":"f30690894927cd0b8abac89b5581f4585a1faca96eaacc7b2af6c4b84bb92dc9"}
Nov 01 05:45:11.464927 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:11.464903    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:45:11.464927 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:11.464926    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:45:11.465170 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:11.464903    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:45:11.465170 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:11.465009    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jh5gk" podUID="7f767e63-7cc7-4b5e-9941-860fe5f4cd62"
Nov 01 05:45:11.465170 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:11.465074    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cwn9s" podUID="a78d89dc-2ed6-4a2e-9558-f54b73a806d2"
Nov 01 05:45:11.465170 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:11.465145    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjh6r" podUID="a18b7694-133e-4663-b8f6-a36f981ab717"
Nov 01 05:45:12.647087 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:12.647049    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nt68j" event={"ID":"c1d9c47e-944e-4112-9889-b01892353659","Type":"ContainerStarted","Data":"aa6858501573c4dfc1d73b595ef950d08986a745dcc8c572d2a95b20d692e0cd"}
Nov 01 05:45:12.647653 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:12.647349    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:45:12.662086 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:12.662052    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:45:12.697252 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:12.697223    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4lm7r"
Nov 01 05:45:12.697402 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:12.697389    2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 05:45:12.697911 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:12.697893    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4lm7r"
Nov 01 05:45:12.922208 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:12.922108    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nt68j" podStartSLOduration=5.904267642 podStartE2EDuration="23.922095039s" totalImagesPullingTime="18.017827397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:12.872207624 +0000 UTC m=+24.928810126" watchObservedRunningTime="2026-11-01 05:45:12.922095039 +0000 UTC m=+24.978697551"
Nov 01 05:45:13.184233 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:13.184157    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:13.465236 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:13.465158    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:45:13.465236 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:13.465184    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:45:13.465420 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:13.465266    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjh6r" podUID="a18b7694-133e-4663-b8f6-a36f981ab717"
Nov 01 05:45:13.465420 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:13.465303    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:45:13.465420 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:13.465382    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jh5gk" podUID="7f767e63-7cc7-4b5e-9941-860fe5f4cd62"
Nov 01 05:45:13.465558 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:13.465431    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cwn9s" podUID="a78d89dc-2ed6-4a2e-9558-f54b73a806d2"
Nov 01 05:45:13.649463 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:13.649432    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:45:13.650025 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:13.649514    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:45:13.664475 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:13.664445    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:45:14.650364 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:14.650317    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:14.791090 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:14.791057    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:15.278025 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:15.277936    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cwn9s"]
Nov 01 05:45:15.278177 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:15.278156    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:45:15.278310 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:15.278276    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cwn9s" podUID="a78d89dc-2ed6-4a2e-9558-f54b73a806d2"
Nov 01 05:45:15.278703 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:15.278680    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jh5gk"]
Nov 01 05:45:15.278813 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:15.278769    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:45:15.278861 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:15.278844    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jh5gk" podUID="7f767e63-7cc7-4b5e-9941-860fe5f4cd62"
Nov 01 05:45:15.327017 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:15.326986    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gjh6r"]
Nov 01 05:45:15.327142 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:15.327094    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:45:15.327198 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:15.327180    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjh6r" podUID="a18b7694-133e-4663-b8f6-a36f981ab717"
Nov 01 05:45:16.464925 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:16.464893    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:45:16.465514 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:16.464891    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:45:16.465514 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:16.465032    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jh5gk" podUID="7f767e63-7cc7-4b5e-9941-860fe5f4cd62"
Nov 01 05:45:16.465514 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:16.465077    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cwn9s" podUID="a78d89dc-2ed6-4a2e-9558-f54b73a806d2"
Nov 01 05:45:17.464958 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:17.464791    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:45:17.465346 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:17.465029    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjh6r" podUID="a18b7694-133e-4663-b8f6-a36f981ab717"
Nov 01 05:45:17.603424 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:17.603394    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:17.742476 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:17.742439    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:18.405760 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:18.405727    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:18.465167 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:18.465136    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:45:18.465604 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:18.465302    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cwn9s" podUID="a78d89dc-2ed6-4a2e-9558-f54b73a806d2"
Nov 01 05:45:18.465604 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:18.465357    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:45:18.465604 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:18.465461    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jh5gk" podUID="7f767e63-7cc7-4b5e-9941-860fe5f4cd62"
Nov 01 05:45:18.555435 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:18.555395    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:18.709587 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:18.709492    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:18.857447 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:18.857405    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:18.997837 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:18.997751    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:19.138977 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:19.138943    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:19.465072 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:19.465039    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:45:19.465231 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:19.465157    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gjh6r" podUID="a18b7694-133e-4663-b8f6-a36f981ab717"
Nov 01 05:45:19.662010 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:19.661974    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:19.802867 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:19.802783    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:19.944701 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:19.944653    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:20.464933 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:20.464896    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:45:20.464933 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:20.464928    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:45:20.465149 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:20.465037    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cwn9s" podUID="a78d89dc-2ed6-4a2e-9558-f54b73a806d2"
Nov 01 05:45:20.465149 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:20.465133    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jh5gk" podUID="7f767e63-7cc7-4b5e-9941-860fe5f4cd62"
Nov 01 05:45:20.553115 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:20.553075    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:20.703764 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:20.703739    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeReady"
Nov 01 05:45:20.703902 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:20.703866    2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 05:45:20.746273 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:20.746201    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:21.408801 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:21.408766    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:45:21.408980 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:21.408815    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs\") pod \"network-metrics-daemon-gjh6r\" (UID: \"a18b7694-133e-4663-b8f6-a36f981ab717\") " pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:45:21.408980 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:21.408936    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:45:21.408980 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:21.408938    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 05:45:21.409069 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:21.409003    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs podName:a18b7694-133e-4663-b8f6-a36f981ab717 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:53.408990165 +0000 UTC m=+65.465592644 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs") pod "network-metrics-daemon-gjh6r" (UID: "a18b7694-133e-4663-b8f6-a36f981ab717") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 05:45:21.409069 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:21.409015    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret podName:7f767e63-7cc7-4b5e-9941-860fe5f4cd62 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:53.409009426 +0000 UTC m=+65.465611906 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret") pod "global-pull-secret-syncer-jh5gk" (UID: "7f767e63-7cc7-4b5e-9941-860fe5f4cd62") : object "kube-system"/"original-pull-secret" not registered
Nov 01 05:45:21.464793 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:21.464767    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:45:21.509127 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:21.509100    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58hcx\" (UniqueName: \"kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx\") pod \"network-check-target-cwn9s\" (UID: \"a78d89dc-2ed6-4a2e-9558-f54b73a806d2\") " pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:45:21.509249 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:21.509233    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 05:45:21.509249 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:21.509248    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 05:45:21.509339 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:21.509257    2583 projected.go:196] Error preparing data for projected volume kube-api-access-58hcx for pod openshift-network-diagnostics/network-check-target-cwn9s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:45:21.509339 ip-10-0-129-213 kubenswrapper[2583]: E1101 05:45:21.509305    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx podName:a78d89dc-2ed6-4a2e-9558-f54b73a806d2 nodeName:}" failed. No retries permitted until 2026-11-01 05:45:53.509291888 +0000 UTC m=+65.565894368 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-58hcx" (UniqueName: "kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx") pod "network-check-target-cwn9s" (UID: "a78d89dc-2ed6-4a2e-9558-f54b73a806d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 05:45:21.517221 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:21.517204    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 05:45:21.517474 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:21.517460    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-87mzr\""
Nov 01 05:45:21.829547 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:21.829445    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:22.233659 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:22.233608    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:22.374697 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:22.374667    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:22.464445 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:22.464416    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:45:22.464597 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:22.464425    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:45:22.527601 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:22.527532    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 05:45:22.527601 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:22.527532    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 05:45:22.527794 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:22.527779    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-v8m4f\""
Nov 01 05:45:22.537287 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:22.537253    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:22.577185 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:22.577167    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 05:45:22.577397 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:22.577383    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-f4d6z\""
Nov 01 05:45:22.841723 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:22.841686    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:23.072639 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:23.072607    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:23.673385 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:23.673363    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xqcsn_1e9a8c0a-32d6-44ef-8a47-f65f6560193d/dns-node-resolver/0.log"
Nov 01 05:45:23.677520 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:23.677496    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:23.863374 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:23.863340    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:24.005357 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:24.005252    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:24.143812 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:24.143778    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:24.201160 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:24.201136    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dnkvk_e5ace06a-0f90-46ef-8a04-fdc0f7180a1e/node-ca/0.log"
Nov 01 05:45:24.313595 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:24.313512    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:24.479744 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:24.479714    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:24.717162 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:24.717132    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:24.857219 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:24.857193    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:25.481127 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:25.481094    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:25.674890 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:25.674856    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:25.846714 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:25.846679    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:26.084989 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:26.084960    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:26.279157 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:26.279071    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:26.879378 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:26.879339    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:27.478032 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:27.477997    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:27.675916 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:27.675883    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:28.074145 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:28.074111    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:28.278999 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:28.278966    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:28.874548 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:28.874514    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:29.476525 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:29.476489    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:29.676111 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:29.676076    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:29.826172 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:29.826090    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:30.075989 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:30.075953    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:30.281396 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:30.281365    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:30.424060 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:30.424026    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:30.881291 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:30.881261    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:31.482361 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:31.482300    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:31.675534 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:31.675501    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:31.819908 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:31.819818    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:32.081569 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:32.081482    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:32.279590 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:32.279552    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:32.478774 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:32.478729    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:34.867317 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:34.867282    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:35.014239 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:35.014198    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:35.153338 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:35.153234    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:35.309506 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:35.309474    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:35.449776 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:35.449685    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:35.589401 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:35.589369    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:35.925670 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:35.925634    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:36.072038 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:36.072003    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:39.019168 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.019137    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vwqd2"]
Nov 01 05:45:39.026370 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.026343    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-qjp2z"]
Nov 01 05:45:39.026544 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.026468    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vwqd2"
Nov 01 05:45:39.028303 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.028281    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kvs75"]
Nov 01 05:45:39.028451 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.028396    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-qjp2z"
Nov 01 05:45:39.030006 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.029987    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kvs75"
Nov 01 05:45:39.046906 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.046882    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-48tvm"]
Nov 01 05:45:39.048407 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.048391    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-48tvm"
Nov 01 05:45:39.051515 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.051495    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nt6ft"]
Nov 01 05:45:39.053560 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.053542    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:45:39.081641 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.081618    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 01 05:45:39.084460 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.084444    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-z8gwd\""
Nov 01 05:45:39.086670 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.086653    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79768fc54c-kxzc9"]
Nov 01 05:45:39.088247 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.088234    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.098940 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.098919    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 01 05:45:39.099064 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.099048    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 01 05:45:39.099104 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.099089    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 05:45:39.100380 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.100367    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 05:45:39.120393 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.120371    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 05:45:39.124108 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.124088    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 05:45:39.126392 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.126368    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:39.129147 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.129126    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-metrics-tls\") pod \"dns-default-kvs75\" (UID: \"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923\") " pod="openshift-dns/dns-default-kvs75"
Nov 01 05:45:39.129245 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.129153    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hhd5\" (UniqueName: \"kubernetes.io/projected/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-kube-api-access-2hhd5\") pod \"dns-default-kvs75\" (UID: \"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923\") " pod="openshift-dns/dns-default-kvs75"
Nov 01 05:45:39.129245 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.129173    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d937ef17-8f48-475a-955a-e0599c40d25c-data-volume\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:45:39.129245 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.129186    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc7de5f7-34ee-46ae-81eb-81717d8f1a52-cert\") pod \"ingress-canary-48tvm\" (UID: \"dc7de5f7-34ee-46ae-81eb-81717d8f1a52\") " pod="openshift-ingress-canary/ingress-canary-48tvm"
Nov 01 05:45:39.129364 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.129302    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-config-volume\") pod \"dns-default-kvs75\" (UID: \"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923\") " pod="openshift-dns/dns-default-kvs75"
Nov 01 05:45:39.129398 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.129365    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3df5a1c7-a784-4bec-81d1-5ee5d36c707c-tmp\") pod \"downloads-6dd789c8b7-qjp2z\" (UID: \"3df5a1c7-a784-4bec-81d1-5ee5d36c707c\") " pod="openshift-console/downloads-6dd789c8b7-qjp2z"
Nov 01 05:45:39.129432 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.129399    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svf57\" (UniqueName: \"kubernetes.io/projected/3df5a1c7-a784-4bec-81d1-5ee5d36c707c-kube-api-access-svf57\") pod \"downloads-6dd789c8b7-qjp2z\" (UID: \"3df5a1c7-a784-4bec-81d1-5ee5d36c707c\") " pod="openshift-console/downloads-6dd789c8b7-qjp2z"
Nov 01 05:45:39.129466 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.129446    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-tmp-dir\") pod \"dns-default-kvs75\" (UID: \"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923\") " pod="openshift-dns/dns-default-kvs75"
Nov 01 05:45:39.129513 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.129487    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlqr8\" (UniqueName: \"kubernetes.io/projected/d937ef17-8f48-475a-955a-e0599c40d25c-kube-api-access-zlqr8\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:45:39.129555 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.129543    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d937ef17-8f48-475a-955a-e0599c40d25c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:45:39.129589 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.129567    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7ttp\" (UniqueName: \"kubernetes.io/projected/dc7de5f7-34ee-46ae-81eb-81717d8f1a52-kube-api-access-w7ttp\") pod \"ingress-canary-48tvm\" (UID: \"dc7de5f7-34ee-46ae-81eb-81717d8f1a52\") " pod="openshift-ingress-canary/ingress-canary-48tvm"
Nov 01 05:45:39.129658 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.129638    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d937ef17-8f48-475a-955a-e0599c40d25c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:45:39.129694 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.129686    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d937ef17-8f48-475a-955a-e0599c40d25c-crio-socket\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:45:39.129726 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.129713    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/06f0eaf3-04d5-492f-9d3a-6b337091fbea-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-vwqd2\" (UID: \"06f0eaf3-04d5-492f-9d3a-6b337091fbea\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vwqd2"
Nov 01 05:45:39.131189 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.131175    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 05:45:39.131225 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.131177    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 05:45:39.191043 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.191011    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 05:45:39.191474 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.191453    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 05:45:39.191572 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.191501    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 05:45:39.191572 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.191535    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-74ptq\""
Nov 01 05:45:39.191572 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.191564    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5zklw\""
Nov 01 05:45:39.191705 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.191453    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 05:45:39.191705 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.191581    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p4xg8\""
Nov 01 05:45:39.191705 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.191454    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 05:45:39.191705 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.191611    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-t2x2d\""
Nov 01 05:45:39.191705 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.191666    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 05:45:39.191705 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.191671    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 05:45:39.191882 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.191760    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vp9l2\""
Nov 01 05:45:39.192841 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.192824    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-qjp2z"]
Nov 01 05:45:39.193377 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.193360    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kvs75"]
Nov 01 05:45:39.210204 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.210180    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nt6ft"]
Nov 01 05:45:39.227975 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.227942    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vwqd2"]
Nov 01 05:45:39.230307 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.230285    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/06f0eaf3-04d5-492f-9d3a-6b337091fbea-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-vwqd2\" (UID: \"06f0eaf3-04d5-492f-9d3a-6b337091fbea\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vwqd2"
Nov 01 05:45:39.230432 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.230317    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-metrics-tls\") pod \"dns-default-kvs75\" (UID: \"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923\") " pod="openshift-dns/dns-default-kvs75"
Nov 01 05:45:39.230432 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.230355    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hhd5\" (UniqueName: \"kubernetes.io/projected/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-kube-api-access-2hhd5\") pod \"dns-default-kvs75\" (UID: \"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923\") " pod="openshift-dns/dns-default-kvs75"
Nov 01 05:45:39.230432 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.230381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d937ef17-8f48-475a-955a-e0599c40d25c-data-volume\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:45:39.230920 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.230508    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc7de5f7-34ee-46ae-81eb-81717d8f1a52-cert\") pod \"ingress-canary-48tvm\" (UID: \"dc7de5f7-34ee-46ae-81eb-81717d8f1a52\") " pod="openshift-ingress-canary/ingress-canary-48tvm"
Nov 01 05:45:39.231007 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.230978    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d937ef17-8f48-475a-955a-e0599c40d25c-data-volume\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:45:39.231061 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.231011    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-config-volume\") pod \"dns-default-kvs75\" (UID: \"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923\") " pod="openshift-dns/dns-default-kvs75"
Nov 01 05:45:39.231061 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.231040    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3df5a1c7-a784-4bec-81d1-5ee5d36c707c-tmp\") pod \"downloads-6dd789c8b7-qjp2z\" (UID: \"3df5a1c7-a784-4bec-81d1-5ee5d36c707c\") " pod="openshift-console/downloads-6dd789c8b7-qjp2z"
Nov 01 05:45:39.231158 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.231080    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svf57\" (UniqueName: \"kubernetes.io/projected/3df5a1c7-a784-4bec-81d1-5ee5d36c707c-kube-api-access-svf57\") pod \"downloads-6dd789c8b7-qjp2z\" (UID: \"3df5a1c7-a784-4bec-81d1-5ee5d36c707c\") " pod="openshift-console/downloads-6dd789c8b7-qjp2z"
Nov 01 05:45:39.231395 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.231374    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-registry-tls\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.231471 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.231423    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-tmp-dir\") pod \"dns-default-kvs75\" (UID: \"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923\") " pod="openshift-dns/dns-default-kvs75"
Nov 01 05:45:39.231471 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.231450    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlqr8\" (UniqueName: \"kubernetes.io/projected/d937ef17-8f48-475a-955a-e0599c40d25c-kube-api-access-zlqr8\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:45:39.231628 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.231479    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-ca-trust-extracted\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.231628 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.231512    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-bound-sa-token\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.231628 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.231548    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d937ef17-8f48-475a-955a-e0599c40d25c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:45:39.231628 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.231564    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-config-volume\") pod \"dns-default-kvs75\" (UID: \"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923\") " pod="openshift-dns/dns-default-kvs75"
Nov 01 05:45:39.231628 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.231580    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-trusted-ca\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.231873 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.231652    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-installation-pull-secrets\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.231873 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.231686    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7ttp\" (UniqueName: \"kubernetes.io/projected/dc7de5f7-34ee-46ae-81eb-81717d8f1a52-kube-api-access-w7ttp\") pod \"ingress-canary-48tvm\" (UID: \"dc7de5f7-34ee-46ae-81eb-81717d8f1a52\") " pod="openshift-ingress-canary/ingress-canary-48tvm"
Nov 01 05:45:39.231873 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.231730    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d937ef17-8f48-475a-955a-e0599c40d25c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:45:39.231873 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.231784    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-image-registry-private-configuration\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.231873 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.231792    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-tmp-dir\") pod \"dns-default-kvs75\" (UID: \"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923\") " pod="openshift-dns/dns-default-kvs75"
Nov 01 05:45:39.232185 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.232162    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3df5a1c7-a784-4bec-81d1-5ee5d36c707c-tmp\") pod \"downloads-6dd789c8b7-qjp2z\" (UID: \"3df5a1c7-a784-4bec-81d1-5ee5d36c707c\") " pod="openshift-console/downloads-6dd789c8b7-qjp2z"
Nov 01 05:45:39.232241 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.232228    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d937ef17-8f48-475a-955a-e0599c40d25c-crio-socket\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:45:39.233063 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.232360    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d937ef17-8f48-475a-955a-e0599c40d25c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:45:39.233063 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.232851    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d937ef17-8f48-475a-955a-e0599c40d25c-crio-socket\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:45:39.233063 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.232890    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-registry-certificates\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.233063 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.233011    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk4fp\" (UniqueName: \"kubernetes.io/projected/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-kube-api-access-wk4fp\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.234364 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.234342    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-metrics-tls\") pod \"dns-default-kvs75\" (UID: \"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923\") " pod="openshift-dns/dns-default-kvs75"
Nov 01 05:45:39.234459 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.234367    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc7de5f7-34ee-46ae-81eb-81717d8f1a52-cert\") pod \"ingress-canary-48tvm\" (UID: \"dc7de5f7-34ee-46ae-81eb-81717d8f1a52\") " pod="openshift-ingress-canary/ingress-canary-48tvm"
Nov 01 05:45:39.234757 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.234735    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-48tvm"]
Nov 01 05:45:39.236051 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.236034    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/06f0eaf3-04d5-492f-9d3a-6b337091fbea-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-vwqd2\" (UID: \"06f0eaf3-04d5-492f-9d3a-6b337091fbea\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vwqd2"
Nov 01 05:45:39.236132 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.236109    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d937ef17-8f48-475a-955a-e0599c40d25c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:45:39.256243 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.256218    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79768fc54c-kxzc9"]
Nov 01 05:45:39.291761 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.291690    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hhd5\" (UniqueName: \"kubernetes.io/projected/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-kube-api-access-2hhd5\") pod \"dns-default-kvs75\" (UID: \"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923\") " pod="openshift-dns/dns-default-kvs75"
Nov 01 05:45:39.292574 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.292547    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlqr8\" (UniqueName: \"kubernetes.io/projected/d937ef17-8f48-475a-955a-e0599c40d25c-kube-api-access-zlqr8\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:45:39.292836 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.292816    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svf57\" (UniqueName: \"kubernetes.io/projected/3df5a1c7-a784-4bec-81d1-5ee5d36c707c-kube-api-access-svf57\") pod \"downloads-6dd789c8b7-qjp2z\" (UID: \"3df5a1c7-a784-4bec-81d1-5ee5d36c707c\") " pod="openshift-console/downloads-6dd789c8b7-qjp2z"
Nov 01 05:45:39.293097 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.293077    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7ttp\" (UniqueName: \"kubernetes.io/projected/dc7de5f7-34ee-46ae-81eb-81717d8f1a52-kube-api-access-w7ttp\") pod \"ingress-canary-48tvm\" (UID: \"dc7de5f7-34ee-46ae-81eb-81717d8f1a52\") " pod="openshift-ingress-canary/ingress-canary-48tvm"
Nov 01 05:45:39.319156 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.319128    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:39.333513 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.333488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-image-registry-private-configuration\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.333636 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.333527    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-registry-certificates\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.333636 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.333552    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk4fp\" (UniqueName: \"kubernetes.io/projected/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-kube-api-access-wk4fp\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.333784 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.333761    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-registry-tls\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.333843 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.333805    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-ca-trust-extracted\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.333843 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.333837    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-bound-sa-token\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.333939 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.333879    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-trusted-ca\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.333939 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.333908    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-installation-pull-secrets\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.334674 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.334652    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-ca-trust-extracted\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.334928 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.334900    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-registry-certificates\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.335195 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.335170    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-trusted-ca\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.338792 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.336670    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-image-registry-private-configuration\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.338792 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.336795    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vwqd2"
Nov 01 05:45:39.338792 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.338419    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-installation-pull-secrets\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.339395 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.339376    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-registry-tls\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.342162 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.342140    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-qjp2z"
Nov 01 05:45:39.348877 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.348855    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kvs75"
Nov 01 05:45:39.357606 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.357585    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-48tvm"
Nov 01 05:45:39.365624 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.365599    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:45:39.400496 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.400418    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk4fp\" (UniqueName: \"kubernetes.io/projected/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-kube-api-access-wk4fp\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.400782 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.400714    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-bound-sa-token\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.650587 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.650551    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:39.697987 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.697941    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:39.728775 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.728576    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vwqd2"]
Nov 01 05:45:39.780249 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.780220    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-qjp2z"]
Nov 01 05:45:39.783231 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.783205    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kvs75"]
Nov 01 05:45:39.783971 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:45:39.783947    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df5a1c7_a784_4bec_81d1_5ee5d36c707c.slice/crio-647768154d03c92133235dfac5255efb626578953a1577b6b4de07a02eb98842 WatchSource:0}: Error finding container 647768154d03c92133235dfac5255efb626578953a1577b6b4de07a02eb98842: Status 404 returned error can't find the container with id 647768154d03c92133235dfac5255efb626578953a1577b6b4de07a02eb98842
Nov 01 05:45:39.786489 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:45:39.786471    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod910a0fc6_8aa8_4b22_b767_b0ecf3b6c923.slice/crio-655999dd3abb0a6ce452d86b95ba89143bf9f1914cb7097198ad4fc28b5012de WatchSource:0}: Error finding container 655999dd3abb0a6ce452d86b95ba89143bf9f1914cb7097198ad4fc28b5012de: Status 404 returned error can't find the container with id 655999dd3abb0a6ce452d86b95ba89143bf9f1914cb7097198ad4fc28b5012de
Nov 01 05:45:39.795150 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.795126    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:39.833553 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.833525    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-48tvm"]
Nov 01 05:45:39.836676 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:39.836652    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nt6ft"]
Nov 01 05:45:39.837433 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:45:39.837401    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc7de5f7_34ee_46ae_81eb_81717d8f1a52.slice/crio-7958929b69dae8fb2c16965e0e68ba60ae49885e0f272cfcb1c28525afcf89e5 WatchSource:0}: Error finding container 7958929b69dae8fb2c16965e0e68ba60ae49885e0f272cfcb1c28525afcf89e5: Status 404 returned error can't find the container with id 7958929b69dae8fb2c16965e0e68ba60ae49885e0f272cfcb1c28525afcf89e5
Nov 01 05:45:39.839795 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:45:39.839774    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd937ef17_8f48_475a_955a_e0599c40d25c.slice/crio-3c168b1451e9ae9c0b21da52fb9636545bbfa9456ed1dc4d957a7aba32dfd671 WatchSource:0}: Error finding container 3c168b1451e9ae9c0b21da52fb9636545bbfa9456ed1dc4d957a7aba32dfd671: Status 404 returned error can't find the container with id 3c168b1451e9ae9c0b21da52fb9636545bbfa9456ed1dc4d957a7aba32dfd671
Nov 01 05:45:40.070077 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:40.070037    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79768fc54c-kxzc9"]
Nov 01 05:45:40.074013 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:45:40.073983    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa8083b6_e023_46fe_bdb3_a9e99079eb3d.slice/crio-b61f46a1b913e6c2c2a59d9b2fc4c61b6d9cb47d64d31e5b21f08b4105d3a218 WatchSource:0}: Error finding container b61f46a1b913e6c2c2a59d9b2fc4c61b6d9cb47d64d31e5b21f08b4105d3a218: Status 404 returned error can't find the container with id b61f46a1b913e6c2c2a59d9b2fc4c61b6d9cb47d64d31e5b21f08b4105d3a218
Nov 01 05:45:40.711083 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:40.711044    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vwqd2" event={"ID":"06f0eaf3-04d5-492f-9d3a-6b337091fbea","Type":"ContainerStarted","Data":"ed91fd3a410fe1b2ee686b94ba8a4c15a4d1f1872bc6366d6a926bb5c3b1cdc5"}
Nov 01 05:45:40.712648 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:40.712618    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kvs75" event={"ID":"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923","Type":"ContainerStarted","Data":"655999dd3abb0a6ce452d86b95ba89143bf9f1914cb7097198ad4fc28b5012de"}
Nov 01 05:45:40.715457 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:40.715318    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nt6ft" event={"ID":"d937ef17-8f48-475a-955a-e0599c40d25c","Type":"ContainerStarted","Data":"d8534282e722b5dfa1f8bf3bf2e15b95181ab1afca6d33fc9c086afe0e55642e"}
Nov 01 05:45:40.715560 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:40.715516    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nt6ft" event={"ID":"d937ef17-8f48-475a-955a-e0599c40d25c","Type":"ContainerStarted","Data":"3c168b1451e9ae9c0b21da52fb9636545bbfa9456ed1dc4d957a7aba32dfd671"}
Nov 01 05:45:40.719427 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:40.719398    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-48tvm" event={"ID":"dc7de5f7-34ee-46ae-81eb-81717d8f1a52","Type":"ContainerStarted","Data":"7958929b69dae8fb2c16965e0e68ba60ae49885e0f272cfcb1c28525afcf89e5"}
Nov 01 05:45:40.721226 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:40.721170    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79768fc54c-kxzc9" event={"ID":"aa8083b6-e023-46fe-bdb3-a9e99079eb3d","Type":"ContainerStarted","Data":"9dbcbdebac599ff5db05be49614fdabe1fba1c00a17d823e2801d46161ea0e21"}
Nov 01 05:45:40.721360 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:40.721245    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79768fc54c-kxzc9" event={"ID":"aa8083b6-e023-46fe-bdb3-a9e99079eb3d","Type":"ContainerStarted","Data":"b61f46a1b913e6c2c2a59d9b2fc4c61b6d9cb47d64d31e5b21f08b4105d3a218"}
Nov 01 05:45:40.721985 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:40.721933    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:45:40.723732 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:40.723691    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-qjp2z" event={"ID":"3df5a1c7-a784-4bec-81d1-5ee5d36c707c","Type":"ContainerStarted","Data":"647768154d03c92133235dfac5255efb626578953a1577b6b4de07a02eb98842"}
Nov 01 05:45:40.842112 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:40.841422    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79768fc54c-kxzc9" podStartSLOduration=1.841402713 podStartE2EDuration="1.841402713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:40.840649112 +0000 UTC m=+52.897251626" watchObservedRunningTime="2026-11-01 05:45:40.841402713 +0000 UTC m=+52.898005216"
Nov 01 05:45:41.729480 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:41.729393    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vwqd2" event={"ID":"06f0eaf3-04d5-492f-9d3a-6b337091fbea","Type":"ContainerStarted","Data":"04e21652c2e280f6f5a2c8619400eaf6032a205fbcf37bf1d4bcadb69b704442"}
Nov 01 05:45:41.729893 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:41.729648    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vwqd2"
Nov 01 05:45:41.733969 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:41.733931    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nt6ft" event={"ID":"d937ef17-8f48-475a-955a-e0599c40d25c","Type":"ContainerStarted","Data":"9cd8aa8670447808dd3ce5d28da301ea56d00d8c1b740feb0b96468330fdc50e"}
Nov 01 05:45:41.736215 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:41.736191    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vwqd2"
Nov 01 05:45:41.847182 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:41.846211    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vwqd2" podStartSLOduration=2.14574951 podStartE2EDuration="3.846190966s" totalImagesPullingTime="1.700441456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:41.844914203 +0000 UTC m=+53.901516706" watchObservedRunningTime="2026-11-01 05:45:41.846190966 +0000 UTC m=+53.902793467"
Nov 01 05:45:42.206085 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:42.205942    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:42.412923 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:42.412861    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:42.561953 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:42.561903    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:42.706511 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:42.706345    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:42.740606 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:42.740521    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kvs75" event={"ID":"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923","Type":"ContainerStarted","Data":"dfb10e9f46f8ab128825e4a5853399bdf32983fe7874b7570452d3f30a83dede"}
Nov 01 05:45:42.740606 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:42.740589    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kvs75" event={"ID":"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923","Type":"ContainerStarted","Data":"dd7634395912dd365ebfa3e36295454c899dec2c3e6560de9c13db3803628d11"}
Nov 01 05:45:42.741115 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:42.741091    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kvs75"
Nov 01 05:45:42.857976 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:42.857527    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-kvs75" podStartSLOduration=2.293113658 podStartE2EDuration="4.857503017s" totalImagesPullingTime="2.564389359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:42.856504855 +0000 UTC m=+54.913107362" watchObservedRunningTime="2026-11-01 05:45:42.857503017 +0000 UTC m=+54.914105519"
Nov 01 05:45:43.745480 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:43.745442    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-48tvm" event={"ID":"dc7de5f7-34ee-46ae-81eb-81717d8f1a52","Type":"ContainerStarted","Data":"c63d38a12b40f9f67de1034d8e6aebc9cdd57c6b2cbdcc3a19a47eac0e358d79"}
Nov 01 05:45:43.747791 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:43.747763    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nt6ft" event={"ID":"d937ef17-8f48-475a-955a-e0599c40d25c","Type":"ContainerStarted","Data":"b180e36ce5d4ef57ed7105be4a77231adb1ca0a5fe08694927863d05511a3c31"}
Nov 01 05:45:43.860108 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:43.859854    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-48tvm" podStartSLOduration=2.069316468 podStartE2EDuration="5.85983301s" totalImagesPullingTime="3.790516542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:43.859031495 +0000 UTC m=+55.915633997" watchObservedRunningTime="2026-11-01 05:45:43.85983301 +0000 UTC m=+55.916435514"
Nov 01 05:45:43.977878 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:43.977822    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nt6ft" podStartSLOduration=1.396066041 podStartE2EDuration="4.977802162s" totalImagesPullingTime="3.581736121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 05:45:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:43.976878971 +0000 UTC m=+56.033481472" watchObservedRunningTime="2026-11-01 05:45:43.977802162 +0000 UTC m=+56.034404665"
Nov 01 05:45:45.665567 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:45.665538    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:45:48.100903 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:48.100842    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:48.249830 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:48.249794    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:48.527172 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:48.527090    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:48.706849 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:48.706808    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:48.856158 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:48.856125    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:49.001898 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.001862    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:49.149270 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.149195    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:49.294257 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.294222    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m5vvc"]
Nov 01 05:45:49.295511 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.295466    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:49.297375 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.297356    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.399253 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.399215    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 05:45:49.399253 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.399249    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f427j\""
Nov 01 05:45:49.399537 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.399223    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 05:45:49.399537 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.399263    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 05:45:49.399537 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.399215    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 05:45:49.399537 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.399289    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 05:45:49.399537 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.399335    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 05:45:49.412692 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.412661    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-accelerators-collector-config\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.412692 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.412697    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-textfile\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.412901 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.412815    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlj5f\" (UniqueName: \"kubernetes.io/projected/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-kube-api-access-zlj5f\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.412957 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.412897    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.412957 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.412930    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-metrics-client-ca\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.413055 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.413020    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-root\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.413055 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.413050    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-tls\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.413157 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.413136    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-sys\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.413207 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.413166    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-wtmp\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.445466 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.445429    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:49.513901 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.513864    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-sys\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.514085 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.513908    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-wtmp\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.514085 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.513943    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-accelerators-collector-config\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.514085 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.513963    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-textfile\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.514085 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.513978    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-sys\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.514085 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.514008    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlj5f\" (UniqueName: \"kubernetes.io/projected/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-kube-api-access-zlj5f\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.514085 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.514073    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.514468 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.514107    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-wtmp\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.514468 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.514106    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-metrics-client-ca\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.514468 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.514162    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-root\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.514468 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.514189    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-tls\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.514668 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.514650    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-root\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.514720 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.514664    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-textfile\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.514720 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.514690    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-metrics-client-ca\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.514973 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.514952    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-accelerators-collector-config\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.516890 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.516861    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-tls\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.517403 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.517378    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.575132 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.575099    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlj5f\" (UniqueName: \"kubernetes.io/projected/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-kube-api-access-zlj5f\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.586629 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.586598    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:49.611465 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.611430    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:45:49.621365 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:45:49.621336    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec642c5b_c364_4faa_94d5_bb5bf32a2cb8.slice/crio-3a75221ba0ae447b96836b7fb9ec29688184b2db03d98b9648de41335a8a6c75 WatchSource:0}: Error finding container 3a75221ba0ae447b96836b7fb9ec29688184b2db03d98b9648de41335a8a6c75: Status 404 returned error can't find the container with id 3a75221ba0ae447b96836b7fb9ec29688184b2db03d98b9648de41335a8a6c75
Nov 01 05:45:49.727144 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.727049    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:49.768578 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:49.768542    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m5vvc" event={"ID":"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8","Type":"ContainerStarted","Data":"3a75221ba0ae447b96836b7fb9ec29688184b2db03d98b9648de41335a8a6c75"}
Nov 01 05:45:50.645491 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:50.645438    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:50.824510 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:50.824476    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:50.963304 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:50.963235    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:51.506315 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.506280    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:51.607302 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.607243    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 05:45:51.612611 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.612571    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.665087 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.665051    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 01 05:45:51.665087 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.665070    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 01 05:45:51.665087 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.665083    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 01 05:45:51.665671 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.665056    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 01 05:45:51.665671 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.665113    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 01 05:45:51.665671 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.665147    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 01 05:45:51.731424 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.731389    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkn77\" (UniqueName: \"kubernetes.io/projected/f4eea300-bfd9-465e-9276-16494acaa883-kube-api-access-bkn77\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.731616 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.731454    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.731616 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.731481    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4eea300-bfd9-465e-9276-16494acaa883-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.731616 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.731563    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f4eea300-bfd9-465e-9276-16494acaa883-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.731820 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.731614    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.731820 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.731696    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.731820 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.731723    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f4eea300-bfd9-465e-9276-16494acaa883-config-out\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.731820 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.731809    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-config-volume\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.731989 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.731857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.731989 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.731947    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.732079 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.731988    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f4eea300-bfd9-465e-9276-16494acaa883-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.732079 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.732013    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-web-config\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.732160 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.732119    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4eea300-bfd9-465e-9276-16494acaa883-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.765838 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.765754    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 05:45:51.768155 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.768129    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 01 05:45:51.768282 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.768155    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-75cn8\""
Nov 01 05:45:51.768282 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.768155    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 01 05:45:51.768282 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.768215    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 01 05:45:51.779704 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.779672    2583 generic.go:356] "Generic (PLEG): container finished" podID="ec642c5b-c364-4faa-94d5-bb5bf32a2cb8" containerID="f3af80acba16092e7853bb55c19aa23f499052f2c83ebed5f2f7e29fd5077ce8" exitCode=0
Nov 01 05:45:51.779847 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.779728    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m5vvc" event={"ID":"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8","Type":"ContainerDied","Data":"f3af80acba16092e7853bb55c19aa23f499052f2c83ebed5f2f7e29fd5077ce8"}
Nov 01 05:45:51.832570 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.832540    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4eea300-bfd9-465e-9276-16494acaa883-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.832667 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.832593    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkn77\" (UniqueName: \"kubernetes.io/projected/f4eea300-bfd9-465e-9276-16494acaa883-kube-api-access-bkn77\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.832667 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.832642    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.832756 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.832677    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4eea300-bfd9-465e-9276-16494acaa883-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.832756 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.832708    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f4eea300-bfd9-465e-9276-16494acaa883-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.832756 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.832747    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.832844 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.832804    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.832844 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.832832    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f4eea300-bfd9-465e-9276-16494acaa883-config-out\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.832898 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.832862    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-config-volume\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.832931 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.832892    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.832981 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.832958    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.833032 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.833006    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f4eea300-bfd9-465e-9276-16494acaa883-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.833064 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.833034    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-web-config\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.833459 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.833434    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4eea300-bfd9-465e-9276-16494acaa883-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.833562 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.833459    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f4eea300-bfd9-465e-9276-16494acaa883-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.834755 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.834429    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4eea300-bfd9-465e-9276-16494acaa883-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.835774 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.835742    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-web-config\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.835854 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.835829    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.835963 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.835938    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.837538 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.837361    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.837538 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.837394    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.837785 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.837756    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f4eea300-bfd9-465e-9276-16494acaa883-config-out\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.838225 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.838206    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.839658 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.839632    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-config-volume\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.842047 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.842015    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f4eea300-bfd9-465e-9276-16494acaa883-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.893547 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.893521    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkn77\" (UniqueName: \"kubernetes.io/projected/f4eea300-bfd9-465e-9276-16494acaa883-kube-api-access-bkn77\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:51.930993 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:51.930946    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:45:52.418129 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:52.418077    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 05:45:52.420384 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:45:52.420356    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4eea300_bfd9_465e_9276_16494acaa883.slice/crio-cc11cb67ba92c1e2a7c27ebe0b6e77b34723fa8217835ad395a71fb8fe3eee8a WatchSource:0}: Error finding container cc11cb67ba92c1e2a7c27ebe0b6e77b34723fa8217835ad395a71fb8fe3eee8a: Status 404 returned error can't find the container with id cc11cb67ba92c1e2a7c27ebe0b6e77b34723fa8217835ad395a71fb8fe3eee8a
Nov 01 05:45:52.784453 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:52.784360    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f4eea300-bfd9-465e-9276-16494acaa883","Type":"ContainerStarted","Data":"cc11cb67ba92c1e2a7c27ebe0b6e77b34723fa8217835ad395a71fb8fe3eee8a"}
Nov 01 05:45:52.787343 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:52.787303    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m5vvc" event={"ID":"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8","Type":"ContainerStarted","Data":"3565cdb0becd3f467990778a6616494e0839f4c2f5b1c998f095f8ff5f8effc0"}
Nov 01 05:45:52.787479 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:52.787376    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m5vvc" event={"ID":"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8","Type":"ContainerStarted","Data":"e60bc3302f3e04331a8dbb46fe9f5b5657812d31b91651ee96f3f1204ddb78ef"}
Nov 01 05:45:52.906093 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:52.906019    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m5vvc" podStartSLOduration=2.461064883 podStartE2EDuration="3.905999652s" totalImagesPullingTime="1.36915896s" totalInitContainerRuntime="75.775809ms" isStatefulPod=true podCreationTimestamp="2026-11-01 05:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:52.904818956 +0000 UTC m=+64.961421459" watchObservedRunningTime="2026-11-01 05:45:52.905999652 +0000 UTC m=+64.962602157"
Nov 01 05:45:53.333411 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.333369    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:53.448342 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.448284    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs\") pod \"network-metrics-daemon-gjh6r\" (UID: \"a18b7694-133e-4663-b8f6-a36f981ab717\") " pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:45:53.448519 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.448420    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:45:53.499720 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.499683    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 05:45:53.499868 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.499808    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 05:45:53.512612 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.512546    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs\") pod \"network-metrics-daemon-gjh6r\" (UID: \"a18b7694-133e-4663-b8f6-a36f981ab717\") " pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:45:53.513819 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.513797    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:45:53.524779 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.524752    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:53.548771 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.548734    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58hcx\" (UniqueName: \"kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx\") pod \"network-check-target-cwn9s\" (UID: \"a78d89dc-2ed6-4a2e-9558-f54b73a806d2\") " pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:45:53.600193 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.600151    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 05:45:53.660262 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.660236    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 05:45:53.666241 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.666213    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:53.672335 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.672294    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58hcx\" (UniqueName: \"kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx\") pod \"network-check-target-cwn9s\" (UID: \"a78d89dc-2ed6-4a2e-9558-f54b73a806d2\") " pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:45:53.698378 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.698344    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-87mzr\""
Nov 01 05:45:53.706977 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.706939    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:45:53.756091 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.756058    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kvs75"
Nov 01 05:45:53.788277 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.788251    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-v8m4f\""
Nov 01 05:45:53.797765 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.797553    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:45:53.804293 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.804265    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:53.806820 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.806802    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-f4d6z\""
Nov 01 05:45:53.811656 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.811631    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:45:53.946605 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.946531    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:53.987243 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.987205    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2brlp"]
Nov 01 05:45:53.990160 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:53.990135    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2brlp"
Nov 01 05:45:54.044610 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.044578    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 01 05:45:54.044610 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.044578    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dj5fv\""
Nov 01 05:45:54.048105 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.047643    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2brlp"]
Nov 01 05:45:54.088171 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.088147    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gjh6r"]
Nov 01 05:45:54.088801 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.088777    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:54.092513 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:45:54.092490    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18b7694_133e_4663_b8f6_a36f981ab717.slice/crio-be6f8f45ab71c357cb49a252f4c1de95650637b9c6931939584d2b2215c10cf3 WatchSource:0}: Error finding container be6f8f45ab71c357cb49a252f4c1de95650637b9c6931939584d2b2215c10cf3: Status 404 returned error can't find the container with id be6f8f45ab71c357cb49a252f4c1de95650637b9c6931939584d2b2215c10cf3
Nov 01 05:45:54.114192 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.114115    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-69999ffccd-cvcrh"]
Nov 01 05:45:54.116472 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.116450    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.155372 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.155316    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stntj\" (UniqueName: \"kubernetes.io/projected/941ae7ae-e26d-4adb-8d74-729e4c038861-kube-api-access-stntj\") pod \"monitoring-plugin-77f4f76594-2brlp\" (UID: \"941ae7ae-e26d-4adb-8d74-729e4c038861\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2brlp"
Nov 01 05:45:54.155569 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.155397    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/941ae7ae-e26d-4adb-8d74-729e4c038861-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2brlp\" (UID: \"941ae7ae-e26d-4adb-8d74-729e4c038861\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2brlp"
Nov 01 05:45:54.168494 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.168463    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 01 05:45:54.168690 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.168672    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 01 05:45:54.168948 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.168927    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9hwbb\""
Nov 01 05:45:54.168948 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.168949    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 01 05:45:54.169126 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.168983    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 01 05:45:54.169126 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.169020    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-12tgr4asi4bd8\""
Nov 01 05:45:54.177356 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.176813    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-69999ffccd-cvcrh"]
Nov 01 05:45:54.186618 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.186590    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jh5gk"]
Nov 01 05:45:54.192334 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:45:54.192294    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f767e63_7cc7_4b5e_9941_860fe5f4cd62.slice/crio-7c35ac7cdc12d77ae99b72c86cbbf2de2ea1de39beb5e86d7b6c5452c71db176 WatchSource:0}: Error finding container 7c35ac7cdc12d77ae99b72c86cbbf2de2ea1de39beb5e86d7b6c5452c71db176: Status 404 returned error can't find the container with id 7c35ac7cdc12d77ae99b72c86cbbf2de2ea1de39beb5e86d7b6c5452c71db176
Nov 01 05:45:54.200818 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.200790    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cwn9s"]
Nov 01 05:45:54.204472 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:45:54.204442    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78d89dc_2ed6_4a2e_9558_f54b73a806d2.slice/crio-6b5955782ac24a8a91697cb38fe4f8d59410ba45199f602bbb9aceab6817cc11 WatchSource:0}: Error finding container 6b5955782ac24a8a91697cb38fe4f8d59410ba45199f602bbb9aceab6817cc11: Status 404 returned error can't find the container with id 6b5955782ac24a8a91697cb38fe4f8d59410ba45199f602bbb9aceab6817cc11
Nov 01 05:45:54.256091 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.256057    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/95fd5853-f502-4076-90a2-19fb99f52e2c-secret-metrics-server-tls\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.256091 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.256096    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhq5k\" (UniqueName: \"kubernetes.io/projected/95fd5853-f502-4076-90a2-19fb99f52e2c-kube-api-access-vhq5k\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.256372 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.256117    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/941ae7ae-e26d-4adb-8d74-729e4c038861-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2brlp\" (UID: \"941ae7ae-e26d-4adb-8d74-729e4c038861\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2brlp"
Nov 01 05:45:54.256372 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.256208    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/95fd5853-f502-4076-90a2-19fb99f52e2c-metrics-server-audit-profiles\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.256372 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.256238    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/95fd5853-f502-4076-90a2-19fb99f52e2c-client-ca-bundle\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.256372 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.256305    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/95fd5853-f502-4076-90a2-19fb99f52e2c-audit-log\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.256372 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.256357    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stntj\" (UniqueName: \"kubernetes.io/projected/941ae7ae-e26d-4adb-8d74-729e4c038861-kube-api-access-stntj\") pod \"monitoring-plugin-77f4f76594-2brlp\" (UID: \"941ae7ae-e26d-4adb-8d74-729e4c038861\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2brlp"
Nov 01 05:45:54.256372 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.256373    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/95fd5853-f502-4076-90a2-19fb99f52e2c-secret-metrics-server-client-certs\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.256660 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.256417    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95fd5853-f502-4076-90a2-19fb99f52e2c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.258971 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.258942    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/941ae7ae-e26d-4adb-8d74-729e4c038861-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2brlp\" (UID: \"941ae7ae-e26d-4adb-8d74-729e4c038861\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2brlp"
Nov 01 05:45:54.317731 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.317694    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stntj\" (UniqueName: \"kubernetes.io/projected/941ae7ae-e26d-4adb-8d74-729e4c038861-kube-api-access-stntj\") pod \"monitoring-plugin-77f4f76594-2brlp\" (UID: \"941ae7ae-e26d-4adb-8d74-729e4c038861\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2brlp"
Nov 01 05:45:54.357095 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.357058    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/95fd5853-f502-4076-90a2-19fb99f52e2c-audit-log\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.357095 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.357099    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/95fd5853-f502-4076-90a2-19fb99f52e2c-secret-metrics-server-client-certs\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.357340 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.357129    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95fd5853-f502-4076-90a2-19fb99f52e2c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.357340 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.357162    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/95fd5853-f502-4076-90a2-19fb99f52e2c-secret-metrics-server-tls\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.357340 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.357285    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhq5k\" (UniqueName: \"kubernetes.io/projected/95fd5853-f502-4076-90a2-19fb99f52e2c-kube-api-access-vhq5k\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.357491 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.357356    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/95fd5853-f502-4076-90a2-19fb99f52e2c-metrics-server-audit-profiles\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.357491 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.357402    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/95fd5853-f502-4076-90a2-19fb99f52e2c-client-ca-bundle\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.357590 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.357558    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/95fd5853-f502-4076-90a2-19fb99f52e2c-audit-log\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.358061 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.358039    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95fd5853-f502-4076-90a2-19fb99f52e2c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.358513 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.358480    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/95fd5853-f502-4076-90a2-19fb99f52e2c-metrics-server-audit-profiles\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.359880 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.359857    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/95fd5853-f502-4076-90a2-19fb99f52e2c-client-ca-bundle\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.360063 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.360045    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/95fd5853-f502-4076-90a2-19fb99f52e2c-secret-metrics-server-client-certs\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.360198 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.360181    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/95fd5853-f502-4076-90a2-19fb99f52e2c-secret-metrics-server-tls\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.418654 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.418579    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhq5k\" (UniqueName: \"kubernetes.io/projected/95fd5853-f502-4076-90a2-19fb99f52e2c-kube-api-access-vhq5k\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.429239 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.429205    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:45:54.603136 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.603100    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2brlp"
Nov 01 05:45:54.705539 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.705462    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:54.796578 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.796544    2583 generic.go:356] "Generic (PLEG): container finished" podID="f4eea300-bfd9-465e-9276-16494acaa883" containerID="5f71a589277cb4dd289414e7212d5b8b60008db7afae72656b26c20e0134450a" exitCode=0
Nov 01 05:45:54.797037 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.796647    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f4eea300-bfd9-465e-9276-16494acaa883","Type":"ContainerDied","Data":"5f71a589277cb4dd289414e7212d5b8b60008db7afae72656b26c20e0134450a"}
Nov 01 05:45:54.798047 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.798008    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cwn9s" event={"ID":"a78d89dc-2ed6-4a2e-9558-f54b73a806d2","Type":"ContainerStarted","Data":"6b5955782ac24a8a91697cb38fe4f8d59410ba45199f602bbb9aceab6817cc11"}
Nov 01 05:45:54.799502 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.799476    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jh5gk" event={"ID":"7f767e63-7cc7-4b5e-9941-860fe5f4cd62","Type":"ContainerStarted","Data":"7c35ac7cdc12d77ae99b72c86cbbf2de2ea1de39beb5e86d7b6c5452c71db176"}
Nov 01 05:45:54.800797 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.800775    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gjh6r" event={"ID":"a18b7694-133e-4663-b8f6-a36f981ab717","Type":"ContainerStarted","Data":"be6f8f45ab71c357cb49a252f4c1de95650637b9c6931939584d2b2215c10cf3"}
Nov 01 05:45:54.808835 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.808812    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-69999ffccd-cvcrh"]
Nov 01 05:45:54.819908 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:45:54.819870    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95fd5853_f502_4076_90a2_19fb99f52e2c.slice/crio-e50e5ec9b15f85b6f2aa2c30ce0d639b58674bb6be0104c1d40b5dacdc10b335 WatchSource:0}: Error finding container e50e5ec9b15f85b6f2aa2c30ce0d639b58674bb6be0104c1d40b5dacdc10b335: Status 404 returned error can't find the container with id e50e5ec9b15f85b6f2aa2c30ce0d639b58674bb6be0104c1d40b5dacdc10b335
Nov 01 05:45:54.842751 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.842716    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:54.979697 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:54.979590    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2brlp"]
Nov 01 05:45:54.985488 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:45:54.985439    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941ae7ae_e26d_4adb_8d74_729e4c038861.slice/crio-b597aaced71271e5cb4b1bbeb4779b21232392778b84a3281a7f298efb137989 WatchSource:0}: Error finding container b597aaced71271e5cb4b1bbeb4779b21232392778b84a3281a7f298efb137989: Status 404 returned error can't find the container with id b597aaced71271e5cb4b1bbeb4779b21232392778b84a3281a7f298efb137989
Nov 01 05:45:55.819399 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:55.819354    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2brlp" event={"ID":"941ae7ae-e26d-4adb-8d74-729e4c038861","Type":"ContainerStarted","Data":"b597aaced71271e5cb4b1bbeb4779b21232392778b84a3281a7f298efb137989"}
Nov 01 05:45:55.826302 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:55.826268    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh" event={"ID":"95fd5853-f502-4076-90a2-19fb99f52e2c","Type":"ContainerStarted","Data":"e50e5ec9b15f85b6f2aa2c30ce0d639b58674bb6be0104c1d40b5dacdc10b335"}
Nov 01 05:45:56.836075 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:56.835399    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gjh6r" event={"ID":"a18b7694-133e-4663-b8f6-a36f981ab717","Type":"ContainerStarted","Data":"5d500d002a4434e1fce9903ad471778921620a9cba49e1a8afa88e047e2b222f"}
Nov 01 05:45:56.836075 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:56.835476    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gjh6r" event={"ID":"a18b7694-133e-4663-b8f6-a36f981ab717","Type":"ContainerStarted","Data":"022e36ba50a6245c3eaa4cc4eba62cd481cd5024a31e59251ffada8ca58ae935"}
Nov 01 05:45:56.952389 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:56.951408    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gjh6r" podStartSLOduration=66.297046743 podStartE2EDuration="1m7.951388841s" totalImagesPullingTime="1.654342098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:56.95058038 +0000 UTC m=+69.007182883" watchObservedRunningTime="2026-11-01 05:45:56.951388841 +0000 UTC m=+69.007991344"
Nov 01 05:45:57.846895 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:57.846861    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2brlp" event={"ID":"941ae7ae-e26d-4adb-8d74-729e4c038861","Type":"ContainerStarted","Data":"c9a327c5c5818ba71a2f07d7c925e73eb8fb12c161ebccaf445c5d064e0b20fc"}
Nov 01 05:45:57.848887 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:57.848866    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2brlp"
Nov 01 05:45:57.850827 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:57.850802    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh" event={"ID":"95fd5853-f502-4076-90a2-19fb99f52e2c","Type":"ContainerStarted","Data":"3026bcc201f74083a38a303a0dfaff8fdcbc5a78515474f1347ea5ccfccc4f5c"}
Nov 01 05:45:57.853655 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:57.853635    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2brlp"
Nov 01 05:45:57.856234 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:57.856180    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f4eea300-bfd9-465e-9276-16494acaa883","Type":"ContainerStarted","Data":"6de0ae75117fddfa787c7ff5deb8d4ad1383a2937cdf7c5fc502728190ea31a1"}
Nov 01 05:45:58.005479 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.005422    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2brlp" podStartSLOduration=2.355829696 podStartE2EDuration="5.005401034s" totalImagesPullingTime="2.649571338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:58.004495566 +0000 UTC m=+70.061098065" watchObservedRunningTime="2026-11-01 05:45:58.005401034 +0000 UTC m=+70.062003537"
Nov 01 05:45:58.069183 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.069132    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:58.239493 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.239429    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh" podStartSLOduration=1.434030235 podStartE2EDuration="4.239405183s" totalImagesPullingTime="2.805374948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:58.236991135 +0000 UTC m=+70.293593638" watchObservedRunningTime="2026-11-01 05:45:58.239405183 +0000 UTC m=+70.296007683"
Nov 01 05:45:58.257271 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.257217    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:58.323350 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.319580    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 05:45:58.330404 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.329434    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.382912 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.382826    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 01 05:45:58.383085 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.383030    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 01 05:45:58.383123 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.383091    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 01 05:45:58.383157 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.383144    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 01 05:45:58.383303 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.383285    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 01 05:45:58.397874 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.397841    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:45:58.431515 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.431480    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dmpoc73dr8cgo\""
Nov 01 05:45:58.431724 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.431559    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 01 05:45:58.431724 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.431613    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vdjzw\""
Nov 01 05:45:58.431724 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.431695    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 01 05:45:58.481820 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.481788    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 01 05:45:58.481971 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.481939    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 01 05:45:58.482054 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.482040    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 01 05:45:58.482122 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.482111    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 01 05:45:58.482181 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.482170    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 01 05:45:58.482234 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.482225    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 01 05:45:58.483161 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.483137    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 01 05:45:58.485558 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.485529    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 05:45:58.494604 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.494569    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.494751 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.494620    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.494751 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.494652    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-web-config\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.494874 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.494776    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.494874 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.494831    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.494966 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.494877    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.494966 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.494936    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.495073 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.494964    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.495073 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.495005    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.495073 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.495036    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.495073 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.495061    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp4hf\" (UniqueName: \"kubernetes.io/projected/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-kube-api-access-rp4hf\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.495256 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.495098    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.495256 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.495124    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.495256 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.495159    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.495256 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.495183    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.495256 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.495212    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.495256 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.495237    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.495586 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.495265    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-config\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.495586 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.495294    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.495586 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.495367    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-config-out\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.596695 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.596648    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.596695 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.596693    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.596928 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.596783    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.596928 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.596830    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.596928 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.596874    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.596928 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.596906    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.597121 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.596931    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp4hf\" (UniqueName: \"kubernetes.io/projected/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-kube-api-access-rp4hf\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.597121 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.596965    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.597121 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.596991    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.597121 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.597030    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.597121 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.597056    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.597121 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.597088    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.597121 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.597115    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.597468 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.597150    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-config\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.597468 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.597178    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.597468 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.597231    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-config-out\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.597468 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.597257    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.597468 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.597289    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.597468 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.597340    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-web-config\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.597468 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.597374    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.597468 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.597449    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.597709 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.597564    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.598243 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.598192    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.598352 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.598248    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.598658 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.598633    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.600598 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.600580    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.601880 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.601849    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.602434 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.602389    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-config-out\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.603217 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.603163    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.603979 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.603889    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.604532 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.604492    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.604532 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.604511    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-config\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.604644 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.604609    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-web-config\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.605135 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.605090    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.605239 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.605156    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.605239 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.605202    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.605506 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.605482    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.607528 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.607502    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.608555 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.608531    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.660868 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.660764    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp4hf\" (UniqueName: \"kubernetes.io/projected/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-kube-api-access-rp4hf\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:58.872353 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.869785    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f4eea300-bfd9-465e-9276-16494acaa883","Type":"ContainerStarted","Data":"a0875ea8e25b658a729adfc968fba494b811b65ce842f2e5a81f04bed34ec7e3"}
Nov 01 05:45:58.872353 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.869915    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f4eea300-bfd9-465e-9276-16494acaa883","Type":"ContainerStarted","Data":"b5e340a9ebe8aa6a315464a8436274221ecec244d501abf23ebb9f13bbdff343"}
Nov 01 05:45:58.872353 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.869973    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f4eea300-bfd9-465e-9276-16494acaa883","Type":"ContainerStarted","Data":"5b5625e8603c063020e3994de5783522f5ec8607b2fc6cc66144e97a4a62c179"}
Nov 01 05:45:58.872353 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.870030    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f4eea300-bfd9-465e-9276-16494acaa883","Type":"ContainerStarted","Data":"ef6cd62ea245b7f75fb4bf3aa2d93e954ea422c80a6b066b48cbb0a6b0ca95bb"}
Nov 01 05:45:58.958216 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:58.958179    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:45:59.485897 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:59.485679    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 05:45:59.489576 ip-10-0-129-213 kubenswrapper[2583]: W1101 05:45:59.489295    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4254b02e_3f9a_4bcc_b711_de1bbb5b3395.slice/crio-8b45d9a43bd1cea46768ba85a72d27e5c6ee0b34f3b7d1db813b95d75a74c225 WatchSource:0}: Error finding container 8b45d9a43bd1cea46768ba85a72d27e5c6ee0b34f3b7d1db813b95d75a74c225: Status 404 returned error can't find the container with id 8b45d9a43bd1cea46768ba85a72d27e5c6ee0b34f3b7d1db813b95d75a74c225
Nov 01 05:45:59.877125 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:59.877027    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-qjp2z" event={"ID":"3df5a1c7-a784-4bec-81d1-5ee5d36c707c","Type":"ContainerStarted","Data":"a0ec8f865a1c6e8336e91225f21f3056386a4225d487379ffb1b14119ccf23f6"}
Nov 01 05:45:59.877125 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:59.877092    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-qjp2z"
Nov 01 05:45:59.880063 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:59.880034    2583 generic.go:356] "Generic (PLEG): container finished" podID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerID="0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03" exitCode=0
Nov 01 05:45:59.880197 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:59.880113    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4254b02e-3f9a-4bcc-b711-de1bbb5b3395","Type":"ContainerDied","Data":"0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03"}
Nov 01 05:45:59.880197 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:59.880184    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4254b02e-3f9a-4bcc-b711-de1bbb5b3395","Type":"ContainerStarted","Data":"8b45d9a43bd1cea46768ba85a72d27e5c6ee0b34f3b7d1db813b95d75a74c225"}
Nov 01 05:45:59.891229 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:59.891112    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-qjp2z"
Nov 01 05:45:59.995218 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:45:59.995154    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-qjp2z" podStartSLOduration=2.889632986 podStartE2EDuration="21.995134775s" totalImagesPullingTime="19.105501789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:45:59.993242665 +0000 UTC m=+72.049845178" watchObservedRunningTime="2026-11-01 05:45:59.995134775 +0000 UTC m=+72.051737277"
Nov 01 05:46:00.890678 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:46:00.890110    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f4eea300-bfd9-465e-9276-16494acaa883","Type":"ContainerStarted","Data":"a967a54f3d73124b9e832286f7d0f79b70882226af4361033325c327605c8de9"}
Nov 01 05:46:00.892865 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:46:00.892473    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cwn9s" event={"ID":"a78d89dc-2ed6-4a2e-9558-f54b73a806d2","Type":"ContainerStarted","Data":"ede47fe1f4d14d5e6b718c2ecd291010e80a22600a9e8866ebefd2eeeb3e3ded"}
Nov 01 05:46:01.020475 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:46:01.020411    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.591427243 podStartE2EDuration="10.020392614s" totalImagesPullingTime="6.409224444s" totalInitContainerRuntime="19.740927ms" isStatefulPod=false podCreationTimestamp="2026-11-01 05:45:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:46:01.019612309 +0000 UTC m=+73.076214852" watchObservedRunningTime="2026-11-01 05:46:01.020392614 +0000 UTC m=+73.076995116"
Nov 01 05:46:01.135736 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:46:01.135670    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cwn9s" podStartSLOduration=66.074350526 podStartE2EDuration="1m12.135649238s" totalImagesPullingTime="6.061298712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:46:01.133863624 +0000 UTC m=+73.190466127" watchObservedRunningTime="2026-11-01 05:46:01.135649238 +0000 UTC m=+73.192251740"
Nov 01 05:46:01.896931 ip-10-0-129-213 kubenswrapper[2583]: I1101 05:46:01.896897    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:46:02.706849 ip-10-0-129-213 systemd[1]: Stopping Kubernetes Kubelet...
Nov 01 05:46:02.721768 ip-10-0-129-213 systemd[1]: kubelet.service: Deactivated successfully.
Nov 01 05:46:02.722010 ip-10-0-129-213 systemd[1]: Stopped Kubernetes Kubelet.
Nov 01 05:46:02.722090 ip-10-0-129-213 systemd[1]: kubelet.service: Consumed 4.174s CPU time.
Nov 01 05:46:02.730100 ip-10-0-129-213 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 05:46:02.839951 ip-10-0-129-213 kubenswrapper[5619]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 05:46:02.839951 ip-10-0-129-213 kubenswrapper[5619]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 05:46:02.839951 ip-10-0-129-213 kubenswrapper[5619]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 05:46:02.839951 ip-10-0-129-213 kubenswrapper[5619]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 05:46:02.842537 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842439    5619 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:46:02.842537 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842534    5619 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:46:02.842537 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842538    5619 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:46:02.842537 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842541    5619 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:46:02.842537 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842544    5619 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:46:02.842537 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842547    5619 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:46:02.842537 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842550    5619 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:46:02.842826 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842553    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:46:02.842826 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842556    5619 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:46:02.842826 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842559    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:46:02.842826 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842561    5619 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:46:02.842826 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842564    5619 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:46:02.842826 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842571    5619 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:46:02.842826 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842574    5619 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:46:02.842826 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842577    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:46:02.842826 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842580    5619 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:46:02.842826 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842583    5619 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:46:02.842826 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842586    5619 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:46:02.842826 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842589    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:46:02.842826 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842591    5619 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:46:02.842826 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842594    5619 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:46:02.842826 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842596    5619 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:46:02.842826 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842599    5619 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:46:02.842826 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842602    5619 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:46:02.842826 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842605    5619 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:46:02.842826 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842608    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:46:02.842826 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842610    5619 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:46:02.843530 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842618    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:46:02.843530 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842621    5619 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:46:02.843530 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842623    5619 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:46:02.843530 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842626    5619 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:46:02.843530 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842629    5619 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:46:02.843530 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842631    5619 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:46:02.843530 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842634    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:46:02.843530 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842636    5619 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:46:02.843530 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842639    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:46:02.843530 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842642    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:46:02.843530 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842644    5619 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:46:02.843530 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842647    5619 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:46:02.843530 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842650    5619 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:46:02.843530 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842655    5619 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:46:02.843530 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842659    5619 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:46:02.843530 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842663    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:46:02.843530 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842667    5619 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:46:02.843530 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842672    5619 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:46:02.843530 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842676    5619 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:46:02.843530 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842680    5619 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:46:02.844204 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842684    5619 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:46:02.844204 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842688    5619 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:46:02.844204 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842692    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:46:02.844204 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842698    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:46:02.844204 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842701    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:46:02.844204 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842704    5619 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:46:02.844204 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842707    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:46:02.844204 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842709    5619 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:46:02.844204 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842712    5619 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:46:02.844204 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842715    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:46:02.844204 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842717    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:46:02.844204 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842720    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:46:02.844204 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842724    5619 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:46:02.844204 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842727    5619 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:46:02.844204 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842730    5619 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:46:02.844204 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842732    5619 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:46:02.844204 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842735    5619 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:46:02.844204 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842738    5619 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:46:02.844204 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842740    5619 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:46:02.844885 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842743    5619 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:46:02.844885 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842745    5619 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:46:02.844885 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842748    5619 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:46:02.844885 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842751    5619 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:46:02.844885 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842754    5619 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:46:02.844885 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842757    5619 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:46:02.844885 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842763    5619 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:46:02.844885 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842767    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:46:02.844885 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842771    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:46:02.844885 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842774    5619 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:46:02.844885 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842777    5619 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:46:02.844885 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842780    5619 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:46:02.844885 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842782    5619 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:46:02.844885 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842785    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:46:02.844885 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842787    5619 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:46:02.844885 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842790    5619 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:46:02.844885 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842793    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:46:02.844885 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842795    5619 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:46:02.844885 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842798    5619 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:46:02.845525 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842800    5619 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:46:02.845525 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842803    5619 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:46:02.845525 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842806    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:46:02.845525 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842808    5619 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:46:02.845525 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842811    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:46:02.845525 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842814    5619 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:46:02.845525 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842818    5619 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:46:02.845525 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842821    5619 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:46:02.845525 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842824    5619 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:46:02.845525 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842827    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:46:02.845525 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842830    5619 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:46:02.845525 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842833    5619 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:46:02.845525 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842836    5619 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:46:02.845525 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842838    5619 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:46:02.845525 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842841    5619 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:46:02.845525 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842846    5619 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:46:02.845525 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842850    5619 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:46:02.845525 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842857    5619 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:46:02.845525 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842862    5619 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:46:02.846152 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842865    5619 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:46:02.846152 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.842869    5619 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:46:02.846152 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843863    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:46:02.846152 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843873    5619 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:46:02.846152 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843876    5619 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:46:02.846152 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843879    5619 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:46:02.846152 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843882    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:46:02.846152 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843884    5619 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:46:02.846152 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843888    5619 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:46:02.846152 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843891    5619 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:46:02.846152 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843893    5619 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:46:02.846152 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843896    5619 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:46:02.846152 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843898    5619 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:46:02.846152 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843901    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:46:02.846152 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843904    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:46:02.846152 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843907    5619 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:46:02.846152 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843909    5619 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:46:02.846152 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843912    5619 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:46:02.846152 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843914    5619 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:46:02.846808 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843918    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:46:02.846808 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843921    5619 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:46:02.846808 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843924    5619 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:46:02.846808 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843927    5619 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:46:02.846808 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843929    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:46:02.846808 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843932    5619 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:46:02.846808 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843937    5619 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:46:02.846808 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843941    5619 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:46:02.846808 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843945    5619 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:46:02.846808 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843949    5619 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:46:02.846808 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843953    5619 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:46:02.846808 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843956    5619 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:46:02.846808 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843958    5619 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:46:02.846808 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843961    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:46:02.846808 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843963    5619 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:46:02.846808 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843967    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:46:02.846808 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843972    5619 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:46:02.846808 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843976    5619 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:46:02.846808 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843980    5619 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:46:02.847472 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843984    5619 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:46:02.847472 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843989    5619 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:46:02.847472 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843993    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:46:02.847472 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.843998    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:46:02.847472 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844002    5619 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:46:02.847472 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844005    5619 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:46:02.847472 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844008    5619 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:46:02.847472 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844011    5619 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:46:02.847472 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844013    5619 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:46:02.847472 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844016    5619 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:46:02.847472 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844018    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:46:02.847472 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844021    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:46:02.847472 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844024    5619 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:46:02.847472 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844027    5619 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:46:02.847472 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844030    5619 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:46:02.847472 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844033    5619 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:46:02.847472 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844035    5619 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:46:02.847472 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844038    5619 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:46:02.847472 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844041    5619 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:46:02.848132 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844044    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:46:02.848132 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844046    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:46:02.848132 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844049    5619 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:46:02.848132 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844051    5619 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:46:02.848132 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844054    5619 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:46:02.848132 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844056    5619 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:46:02.848132 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844059    5619 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:46:02.848132 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844061    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:46:02.848132 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844064    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:46:02.848132 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844067    5619 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:46:02.848132 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844070    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:46:02.848132 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844072    5619 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:46:02.848132 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844075    5619 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:46:02.848132 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844078    5619 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:46:02.848132 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844081    5619 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:46:02.848132 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844084    5619 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:46:02.848132 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844087    5619 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:46:02.848132 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844090    5619 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:46:02.848132 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844092    5619 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:46:02.848132 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844095    5619 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:46:02.848902 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844098    5619 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:46:02.848902 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844100    5619 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:46:02.848902 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844103    5619 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:46:02.848902 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844105    5619 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:46:02.848902 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844108    5619 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:46:02.848902 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844111    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:46:02.848902 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844115    5619 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:46:02.848902 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844119    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:46:02.848902 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844123    5619 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:46:02.848902 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844126    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:46:02.848902 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844129    5619 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:46:02.848902 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844131    5619 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:46:02.848902 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844134    5619 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:46:02.848902 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844136    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:46:02.848902 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844139    5619 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:46:02.848902 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844141    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:46:02.848902 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844144    5619 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:46:02.848902 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844147    5619 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:46:02.848902 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844149    5619 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:46:02.848902 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844152    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:46:02.849615 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844155    5619 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:46:02.849615 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844159    5619 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:46:02.849615 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844163    5619 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:46:02.849615 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844167    5619 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:46:02.849615 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844173    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:46:02.849615 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844178    5619 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:46:02.849615 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844181    5619 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:46:02.849615 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844184    5619 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:46:02.849615 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844187    5619 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:46:02.849615 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844190    5619 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:46:02.849615 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844194    5619 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:46:02.849615 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844796    5619 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:46:02.849615 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844806    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:46:02.849615 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844811    5619 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:46:02.849615 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844815    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:46:02.849615 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844820    5619 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:46:02.849615 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844824    5619 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:46:02.849615 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844829    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:46:02.849615 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844833    5619 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:46:02.849615 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844838    5619 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:46:02.850429 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844842    5619 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:46:02.850429 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844847    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:46:02.850429 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844851    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:46:02.850429 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844856    5619 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:46:02.850429 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844860    5619 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:46:02.850429 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844863    5619 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:46:02.850429 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844867    5619 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:46:02.850429 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844871    5619 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:46:02.850429 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844876    5619 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:46:02.850429 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844881    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:46:02.850429 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844886    5619 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:46:02.850429 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844890    5619 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:46:02.850429 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844898    5619 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:46:02.850429 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844902    5619 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:46:02.850429 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844907    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:46:02.850429 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844912    5619 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:46:02.850429 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844916    5619 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:46:02.850429 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844921    5619 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:46:02.850429 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844925    5619 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:46:02.850429 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844930    5619 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:46:02.851092 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844933    5619 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:46:02.851092 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844936    5619 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:46:02.851092 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844939    5619 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:46:02.851092 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844941    5619 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:46:02.851092 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844944    5619 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:46:02.851092 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844947    5619 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:46:02.851092 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844951    5619 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:46:02.851092 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844956    5619 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:46:02.851092 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844960    5619 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:46:02.851092 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844965    5619 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:46:02.851092 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844969    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:46:02.851092 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844973    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:46:02.851092 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844977    5619 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:46:02.851092 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844981    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:46:02.851092 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844986    5619 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:46:02.851092 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844990    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:46:02.851092 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844995    5619 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:46:02.851092 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.844998    5619 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:46:02.851092 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845001    5619 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:46:02.851751 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845004    5619 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:46:02.851751 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845006    5619 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:46:02.851751 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845009    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:46:02.851751 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845013    5619 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:46:02.851751 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845018    5619 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:46:02.851751 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845025    5619 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:46:02.851751 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845030    5619 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:46:02.851751 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845034    5619 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:46:02.851751 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845038    5619 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:46:02.851751 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845043    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:46:02.851751 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845047    5619 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:46:02.851751 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845050    5619 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:46:02.851751 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845054    5619 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:46:02.851751 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845057    5619 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:46:02.851751 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845059    5619 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:46:02.851751 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845062    5619 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:46:02.851751 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845065    5619 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:46:02.851751 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845068    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:46:02.851751 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845070    5619 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:46:02.852383 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845073    5619 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:46:02.852383 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845076    5619 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:46:02.852383 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845079    5619 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:46:02.852383 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845084    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:46:02.852383 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845088    5619 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:46:02.852383 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845093    5619 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:46:02.852383 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845097    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:46:02.852383 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845102    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:46:02.852383 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845106    5619 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:46:02.852383 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845109    5619 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:46:02.852383 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845112    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:46:02.852383 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845115    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:46:02.852383 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845117    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:46:02.852383 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845120    5619 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:46:02.852383 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845122    5619 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:46:02.852383 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845125    5619 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:46:02.852383 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845127    5619 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:46:02.852383 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845130    5619 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:46:02.852383 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845135    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:46:02.852383 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845137    5619 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:46:02.853104 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845140    5619 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:46:02.853104 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845143    5619 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:46:02.853104 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845145    5619 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:46:02.853104 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845148    5619 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:46:02.853104 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845151    5619 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:46:02.853104 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845158    5619 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:46:02.853104 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845162    5619 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:46:02.853104 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845165    5619 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:46:02.853104 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845168    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:46:02.853104 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845172    5619 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:46:02.853104 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845174    5619 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:46:02.853104 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845177    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:46:02.853104 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845181    5619 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:46:02.853104 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845186    5619 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:46:02.853104 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845190    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:46:02.853104 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845194    5619 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:46:02.853104 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845198    5619 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:46:02.853104 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845203    5619 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:46:02.853104 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.845207    5619 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:46:02.853104 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845343    5619 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845353    5619 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845361    5619 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845367    5619 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845371    5619 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845375    5619 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845379    5619 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845383    5619 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845389    5619 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845394    5619 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845398    5619 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845404    5619 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845410    5619 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845415    5619 flags.go:64] FLAG: --cgroup-root=""
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845419    5619 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845422    5619 flags.go:64] FLAG: --client-ca-file=""
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845425    5619 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845429    5619 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845435    5619 flags.go:64] FLAG: --cluster-domain=""
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845439    5619 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845445    5619 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845450    5619 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845455    5619 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845461    5619 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 05:46:02.853797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845466    5619 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845470    5619 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845473    5619 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845477    5619 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845480    5619 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845484    5619 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845488    5619 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845492    5619 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845499    5619 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845504    5619 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845509    5619 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845512    5619 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845515    5619 flags.go:64] FLAG: --enable-server="true"
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845518    5619 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845524    5619 flags.go:64] FLAG: --event-burst="100"
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845527    5619 flags.go:64] FLAG: --event-qps="50"
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845530    5619 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845534    5619 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845537    5619 flags.go:64] FLAG: --eviction-hard=""
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845543    5619 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845548    5619 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845553    5619 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845558    5619 flags.go:64] FLAG: --eviction-soft=""
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845563    5619 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845568    5619 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 05:46:02.854677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845599    5619 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845608    5619 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845613    5619 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845616    5619 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845619    5619 flags.go:64] FLAG: --feature-gates=""
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845624    5619 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845627    5619 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845631    5619 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845634    5619 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845638    5619 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845641    5619 flags.go:64] FLAG: --help="false"
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845644    5619 flags.go:64] FLAG: --hostname-override="ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845648    5619 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845651    5619 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845654    5619 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845657    5619 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845661    5619 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845664    5619 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845667    5619 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845670    5619 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845673    5619 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845677    5619 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845680    5619 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845683    5619 flags.go:64] FLAG: --kube-reserved=""
Nov 01 05:46:02.855582 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845686    5619 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845689    5619 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845694    5619 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845697    5619 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845700    5619 flags.go:64] FLAG: --lock-file=""
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845703    5619 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845706    5619 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845709    5619 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845716    5619 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845720    5619 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845723    5619 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845726    5619 flags.go:64] FLAG: --logging-format="text"
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845729    5619 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845733    5619 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845736    5619 flags.go:64] FLAG: --manifest-url=""
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845741    5619 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845748    5619 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845753    5619 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845759    5619 flags.go:64] FLAG: --max-pods="110"
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845764    5619 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845769    5619 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845772    5619 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845775    5619 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845778    5619 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845783    5619 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 05:46:02.856500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845788    5619 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845799    5619 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845804    5619 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845809    5619 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845814    5619 flags.go:64] FLAG: --pod-cidr=""
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845818    5619 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845823    5619 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845828    5619 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845831    5619 flags.go:64] FLAG: --port="10250"
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845836    5619 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845841    5619 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-000ff5393d2565801"
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845846    5619 flags.go:64] FLAG: --qos-reserved=""
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845850    5619 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845855    5619 flags.go:64] FLAG: --register-node="true"
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845861    5619 flags.go:64] FLAG: --register-with-taints=""
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845867    5619 flags.go:64] FLAG: --registry-burst="10"
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845874    5619 flags.go:64] FLAG: --registry-qps="5"
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845879    5619 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845883    5619 flags.go:64] FLAG: --reserved-memory=""
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845887    5619 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845891    5619 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845894    5619 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845897    5619 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845900    5619 flags.go:64] FLAG: --runonce="false"
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845903    5619 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845908    5619 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845913    5619 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 05:46:02.857482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845917    5619 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845921    5619 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845924    5619 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845928    5619 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845932    5619 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845937    5619 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845941    5619 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845947    5619 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845952    5619 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845961    5619 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845964    5619 flags.go:64] FLAG: --system-cgroups=""
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845967    5619 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845973    5619 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845976    5619 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845987    5619 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845994    5619 flags.go:64] FLAG: --tls-min-version=""
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.845999    5619 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.846004    5619 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.846008    5619 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.846013    5619 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.846018    5619 flags.go:64] FLAG: --v="2"
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.846027    5619 flags.go:64] FLAG: --version="false"
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.846033    5619 flags.go:64] FLAG: --vmodule=""
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.846041    5619 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.846048    5619 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 05:46:02.858493 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846240    5619 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:46:02.859382 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846246    5619 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:46:02.859382 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846251    5619 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:46:02.859382 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846258    5619 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:46:02.859382 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846263    5619 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:46:02.859382 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846268    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:46:02.859382 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846273    5619 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:46:02.859382 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846278    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:46:02.859382 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846281    5619 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:46:02.859382 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846284    5619 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:46:02.859382 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846286    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:46:02.859382 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846289    5619 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:46:02.859382 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846292    5619 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:46:02.859382 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846295    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:46:02.859382 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846298    5619 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:46:02.859382 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846301    5619 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:46:02.859382 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846303    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:46:02.859382 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846306    5619 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:46:02.859382 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846309    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:46:02.859382 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846311    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:46:02.860190 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846314    5619 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:46:02.860190 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846332    5619 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:46:02.860190 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846336    5619 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:46:02.860190 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846339    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:46:02.860190 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846342    5619 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:46:02.860190 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846344    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:46:02.860190 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846347    5619 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:46:02.860190 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846350    5619 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:46:02.860190 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846358    5619 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:46:02.860190 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846362    5619 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:46:02.860190 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846367    5619 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:46:02.860190 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846371    5619 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:46:02.860190 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846376    5619 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:46:02.860190 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846381    5619 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:46:02.860190 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846385    5619 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:46:02.860190 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846389    5619 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:46:02.860190 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846394    5619 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:46:02.860190 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846398    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:46:02.860190 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846401    5619 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:46:02.860190 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846404    5619 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:46:02.860966 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846406    5619 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:46:02.860966 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846409    5619 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:46:02.860966 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846412    5619 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:46:02.860966 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846417    5619 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:46:02.860966 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846421    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:46:02.860966 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846426    5619 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:46:02.860966 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846430    5619 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:46:02.860966 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846435    5619 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:46:02.860966 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846440    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:46:02.860966 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846444    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:46:02.860966 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846449    5619 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:46:02.860966 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846454    5619 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:46:02.860966 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846458    5619 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:46:02.860966 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846463    5619 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:46:02.860966 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846468    5619 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:46:02.860966 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846473    5619 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:46:02.860966 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846477    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:46:02.860966 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846480    5619 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:46:02.860966 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846484    5619 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:46:02.860966 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846489    5619 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:46:02.861671 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846494    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:46:02.861671 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846497    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:46:02.861671 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846502    5619 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:46:02.861671 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846507    5619 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:46:02.861671 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846510    5619 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:46:02.861671 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846513    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:46:02.861671 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846516    5619 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:46:02.861671 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846519    5619 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:46:02.861671 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846522    5619 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:46:02.861671 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846524    5619 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:46:02.861671 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846528    5619 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:46:02.861671 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846532    5619 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:46:02.861671 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846537    5619 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:46:02.861671 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846542    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:46:02.861671 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846546    5619 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:46:02.861671 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846551    5619 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:46:02.861671 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846556    5619 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:46:02.861671 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846562    5619 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:46:02.861671 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846566    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:46:02.862396 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846571    5619 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:46:02.862396 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846575    5619 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:46:02.862396 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846580    5619 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:46:02.862396 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846584    5619 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:46:02.862396 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846589    5619 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:46:02.862396 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846593    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:46:02.862396 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846597    5619 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:46:02.862396 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846601    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:46:02.862396 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846604    5619 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:46:02.862396 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846609    5619 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:46:02.862396 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846613    5619 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:46:02.862396 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846617    5619 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:46:02.862396 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846621    5619 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:46:02.862396 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846627    5619 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:46:02.862396 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846631    5619 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:46:02.862396 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846636    5619 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:46:02.862396 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846640    5619 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:46:02.862396 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846645    5619 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:46:02.862396 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846648    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:46:02.863081 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846652    5619 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:46:02.863081 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846655    5619 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:46:02.863081 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846658    5619 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:46:02.863081 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846662    5619 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:46:02.863081 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846666    5619 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:46:02.863081 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846670    5619 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:46:02.863081 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846675    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:46:02.863081 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846680    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:46:02.863081 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.846716    5619 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 05:46:02.863081 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846779    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:46:02.863081 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846786    5619 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:46:02.863081 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846789    5619 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:46:02.863081 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846791    5619 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:46:02.863081 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846794    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:46:02.863081 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846796    5619 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:46:02.863081 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846799    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:46:02.863692 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846804    5619 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:46:02.863692 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846808    5619 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:46:02.863692 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846813    5619 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:46:02.863692 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846818    5619 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:46:02.863692 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846822    5619 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:46:02.863692 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846827    5619 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:46:02.863692 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846831    5619 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:46:02.863692 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846835    5619 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:46:02.863692 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846838    5619 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:46:02.863692 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846841    5619 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:46:02.863692 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846845    5619 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:46:02.863692 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846848    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:46:02.863692 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846850    5619 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:46:02.863692 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846853    5619 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:46:02.863692 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846856    5619 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:46:02.863692 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846858    5619 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:46:02.863692 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846861    5619 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:46:02.863692 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846864    5619 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:46:02.863692 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846867    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:46:02.863692 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846871    5619 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:46:02.864658 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846874    5619 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:46:02.864658 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846876    5619 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:46:02.864658 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846879    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:46:02.864658 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846883    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:46:02.864658 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846886    5619 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:46:02.864658 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846889    5619 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:46:02.864658 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846891    5619 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:46:02.864658 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846894    5619 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:46:02.864658 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846899    5619 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:46:02.864658 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846903    5619 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:46:02.864658 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846909    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:46:02.864658 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846914    5619 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:46:02.864658 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846919    5619 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:46:02.864658 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846924    5619 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:46:02.864658 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846927    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:46:02.864658 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846929    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:46:02.864658 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846932    5619 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:46:02.864658 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846935    5619 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:46:02.864658 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846938    5619 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:46:02.865504 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846940    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:46:02.865504 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846943    5619 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:46:02.865504 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846948    5619 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:46:02.865504 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846955    5619 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:46:02.865504 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846959    5619 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:46:02.865504 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846966    5619 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:46:02.865504 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846971    5619 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:46:02.865504 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846975    5619 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:46:02.865504 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846980    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:46:02.865504 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846985    5619 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:46:02.865504 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846990    5619 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:46:02.865504 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.846997    5619 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:46:02.865504 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847000    5619 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:46:02.865504 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847003    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:46:02.865504 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847006    5619 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:46:02.865504 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847008    5619 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:46:02.865504 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847012    5619 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:46:02.865504 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847016    5619 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:46:02.865504 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847020    5619 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:46:02.866385 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847025    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:46:02.866385 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847029    5619 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:46:02.866385 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847033    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:46:02.866385 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847037    5619 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:46:02.866385 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847042    5619 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:46:02.866385 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847046    5619 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:46:02.866385 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847049    5619 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:46:02.866385 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847052    5619 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:46:02.866385 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847055    5619 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:46:02.866385 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847058    5619 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:46:02.866385 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847061    5619 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:46:02.866385 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847063    5619 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:46:02.866385 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847066    5619 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:46:02.866385 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847069    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:46:02.866385 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847071    5619 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:46:02.866385 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847074    5619 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:46:02.866385 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847099    5619 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:46:02.866385 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847107    5619 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:46:02.866385 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847112    5619 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:46:02.867165 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847118    5619 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:46:02.867165 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847122    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:46:02.867165 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847126    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:46:02.867165 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847131    5619 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:46:02.867165 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847136    5619 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:46:02.867165 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847142    5619 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:46:02.867165 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847150    5619 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:46:02.867165 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847155    5619 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:46:02.867165 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847159    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:46:02.867165 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847162    5619 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:46:02.867165 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847167    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:46:02.867165 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847171    5619 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:46:02.867165 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847175    5619 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:46:02.867165 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847178    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:46:02.867165 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847181    5619 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:46:02.867165 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847184    5619 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:46:02.867165 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847187    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:46:02.867165 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847192    5619 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:46:02.867165 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847196    5619 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:46:02.868014 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847201    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:46:02.868014 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847204    5619 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:46:02.868014 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.847207    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:46:02.868014 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.851432    5619 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 05:46:02.868014 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.851447    5619 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 05:46:02.868014 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.851468    5619 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 05:46:02.868014 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.851475    5619 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 05:46:02.868014 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851556    5619 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:46:02.868014 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851565    5619 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:46:02.868014 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851570    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:46:02.868014 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851576    5619 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:46:02.868014 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851581    5619 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:46:02.868014 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851585    5619 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:46:02.868014 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851589    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:46:02.868014 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851592    5619 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:46:02.868014 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851594    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:46:02.868014 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851597    5619 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:46:02.868014 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851600    5619 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:46:02.868014 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851603    5619 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:46:02.868835 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851607    5619 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:46:02.868835 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851610    5619 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:46:02.868835 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851613    5619 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:46:02.868835 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851615    5619 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:46:02.868835 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851618    5619 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:46:02.868835 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851621    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:46:02.868835 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851624    5619 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:46:02.868835 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851626    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:46:02.868835 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851629    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:46:02.868835 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851632    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:46:02.868835 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851636    5619 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:46:02.868835 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851640    5619 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:46:02.868835 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851644    5619 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:46:02.868835 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851648    5619 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:46:02.868835 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851654    5619 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:46:02.868835 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851660    5619 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:46:02.868835 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851665    5619 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:46:02.868835 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851669    5619 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:46:02.868835 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851674    5619 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:46:02.868835 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851678    5619 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:46:02.869690 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851682    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:46:02.869690 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851686    5619 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:46:02.869690 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851691    5619 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:46:02.869690 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851696    5619 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:46:02.869690 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851700    5619 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:46:02.869690 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851705    5619 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:46:02.869690 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851710    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:46:02.869690 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851715    5619 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:46:02.869690 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851717    5619 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:46:02.869690 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851720    5619 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:46:02.869690 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851723    5619 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:46:02.869690 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851727    5619 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:46:02.869690 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851732    5619 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:46:02.869690 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851737    5619 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:46:02.869690 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851742    5619 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:46:02.869690 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851747    5619 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:46:02.869690 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851750    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:46:02.869690 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851753    5619 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:46:02.869690 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851756    5619 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:46:02.870494 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851759    5619 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:46:02.870494 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851762    5619 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:46:02.870494 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851764    5619 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:46:02.870494 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851768    5619 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:46:02.870494 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851770    5619 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:46:02.870494 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851773    5619 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:46:02.870494 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851776    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:46:02.870494 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851778    5619 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:46:02.870494 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851781    5619 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:46:02.870494 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851784    5619 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:46:02.870494 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851786    5619 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:46:02.870494 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851789    5619 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:46:02.870494 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851792    5619 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:46:02.870494 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851795    5619 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:46:02.870494 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851798    5619 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:46:02.870494 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851800    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:46:02.870494 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851803    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:46:02.870494 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851806    5619 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:46:02.870494 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851810    5619 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:46:02.870494 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851814    5619 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:46:02.871360 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851818    5619 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:46:02.871360 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851822    5619 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:46:02.871360 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851827    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:46:02.871360 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851832    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:46:02.871360 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851836    5619 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:46:02.871360 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851840    5619 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:46:02.871360 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851842    5619 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:46:02.871360 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851847    5619 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:46:02.871360 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851852    5619 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:46:02.871360 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851856    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:46:02.871360 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851861    5619 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:46:02.871360 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851865    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:46:02.871360 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851869    5619 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:46:02.871360 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851873    5619 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:46:02.871360 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851877    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:46:02.871360 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851879    5619 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:46:02.871360 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851882    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:46:02.871360 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851885    5619 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:46:02.871360 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851888    5619 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:46:02.871360 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851892    5619 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:46:02.872206 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851896    5619 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:46:02.872206 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851901    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:46:02.872206 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851906    5619 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:46:02.872206 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851910    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:46:02.872206 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851914    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:46:02.872206 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851918    5619 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:46:02.872206 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851921    5619 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:46:02.872206 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851924    5619 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:46:02.872206 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851926    5619 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:46:02.872206 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851929    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:46:02.872206 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851933    5619 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:46:02.872206 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851937    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:46:02.872206 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851941    5619 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:46:02.872206 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851944    5619 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:46:02.872206 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.851947    5619 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:46:02.872844 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.851977    5619 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 05:46:02.872844 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852156    5619 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:46:02.872844 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852162    5619 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:46:02.872844 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852166    5619 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:46:02.872844 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852168    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:46:02.872844 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852172    5619 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:46:02.872844 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852175    5619 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:46:02.872844 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852178    5619 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:46:02.872844 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852181    5619 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:46:02.872844 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852184    5619 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:46:02.872844 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852186    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:46:02.872844 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852189    5619 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:46:02.872844 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852192    5619 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:46:02.872844 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852195    5619 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:46:02.872844 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852197    5619 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:46:02.872844 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852200    5619 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:46:02.873645 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852202    5619 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:46:02.873645 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852205    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:46:02.873645 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852208    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:46:02.873645 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852211    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:46:02.873645 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852213    5619 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:46:02.873645 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852220    5619 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:46:02.873645 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852224    5619 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:46:02.873645 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852229    5619 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:46:02.873645 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852234    5619 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:46:02.873645 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852238    5619 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:46:02.873645 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852243    5619 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:46:02.873645 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852247    5619 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:46:02.873645 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852250    5619 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:46:02.873645 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852253    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:46:02.873645 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852255    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:46:02.873645 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852260    5619 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:46:02.873645 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852264    5619 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:46:02.873645 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852267    5619 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:46:02.873645 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852269    5619 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:46:02.874541 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852276    5619 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:46:02.874541 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852281    5619 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:46:02.874541 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852286    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:46:02.874541 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852291    5619 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:46:02.874541 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852296    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:46:02.874541 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852301    5619 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:46:02.874541 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852306    5619 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:46:02.874541 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852309    5619 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:46:02.874541 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852312    5619 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:46:02.874541 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852316    5619 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:46:02.874541 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852340    5619 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:46:02.874541 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852344    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:46:02.874541 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852349    5619 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:46:02.874541 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852353    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:46:02.874541 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852359    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:46:02.874541 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852365    5619 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:46:02.874541 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852370    5619 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:46:02.874541 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852375    5619 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:46:02.874541 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852382    5619 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:46:02.875428 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852385    5619 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:46:02.875428 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852388    5619 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:46:02.875428 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852391    5619 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:46:02.875428 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852394    5619 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:46:02.875428 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852397    5619 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:46:02.875428 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852402    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:46:02.875428 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852406    5619 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:46:02.875428 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852410    5619 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:46:02.875428 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852414    5619 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:46:02.875428 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852420    5619 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:46:02.875428 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852424    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:46:02.875428 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852427    5619 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:46:02.875428 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852430    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:46:02.875428 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852433    5619 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:46:02.875428 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852435    5619 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:46:02.875428 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852438    5619 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:46:02.875428 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852441    5619 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:46:02.875428 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852444    5619 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:46:02.875428 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852447    5619 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:46:02.875428 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852450    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:46:02.876415 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852453    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:46:02.876415 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852455    5619 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:46:02.876415 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852458    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:46:02.876415 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852463    5619 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:46:02.876415 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852467    5619 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:46:02.876415 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852471    5619 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:46:02.876415 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852474    5619 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:46:02.876415 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852477    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:46:02.876415 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852479    5619 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:46:02.876415 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852482    5619 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:46:02.876415 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852484    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:46:02.876415 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852489    5619 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:46:02.876415 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852491    5619 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:46:02.876415 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852494    5619 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:46:02.876415 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852497    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:46:02.876415 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852500    5619 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:46:02.876415 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852503    5619 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:46:02.876415 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852505    5619 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:46:02.876415 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852509    5619 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:46:02.876415 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852514    5619 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:46:02.877515 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852518    5619 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:46:02.877515 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852523    5619 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:46:02.877515 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852528    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:46:02.877515 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852531    5619 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:46:02.877515 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852535    5619 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:46:02.877515 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852539    5619 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:46:02.877515 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852544    5619 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:46:02.877515 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852549    5619 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:46:02.877515 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852553    5619 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:46:02.877515 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852556    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:46:02.877515 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852559    5619 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:46:02.877515 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852563    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:46:02.877515 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852568    5619 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:46:02.877515 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.852606    5619 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 05:46:02.877515 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852649    5619 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 05:46:02.877515 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852654    5619 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 05:46:02.878311 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852657    5619 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 05:46:02.878311 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852660    5619 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 05:46:02.878311 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852663    5619 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 05:46:02.878311 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852665    5619 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 05:46:02.878311 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852669    5619 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 05:46:02.878311 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852672    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 05:46:02.878311 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852676    5619 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 05:46:02.878311 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852680    5619 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 05:46:02.878311 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852686    5619 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 05:46:02.878311 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852690    5619 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 05:46:02.878311 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852694    5619 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 05:46:02.878311 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852697    5619 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 05:46:02.878311 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852700    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 05:46:02.878311 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852703    5619 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 05:46:02.878311 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852706    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 05:46:02.878311 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852709    5619 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 05:46:02.878311 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852713    5619 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 05:46:02.878311 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852717    5619 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 05:46:02.878311 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852722    5619 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 05:46:02.878311 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852726    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 05:46:02.879197 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852729    5619 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 05:46:02.879197 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852732    5619 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 05:46:02.879197 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852734    5619 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 05:46:02.879197 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852737    5619 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 05:46:02.879197 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852740    5619 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 05:46:02.879197 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852743    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 05:46:02.879197 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852747    5619 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 05:46:02.879197 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852752    5619 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 05:46:02.879197 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852756    5619 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 05:46:02.879197 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852760    5619 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 05:46:02.879197 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852765    5619 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 05:46:02.879197 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852769    5619 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 05:46:02.879197 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852773    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 05:46:02.879197 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852776    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 05:46:02.879197 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852779    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 05:46:02.879197 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852781    5619 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 05:46:02.879197 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852784    5619 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 05:46:02.879197 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852787    5619 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 05:46:02.879197 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852789    5619 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 05:46:02.880030 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852792    5619 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 05:46:02.880030 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852796    5619 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 05:46:02.880030 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852799    5619 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 05:46:02.880030 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852801    5619 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 05:46:02.880030 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852804    5619 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 05:46:02.880030 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852807    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 05:46:02.880030 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852809    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 05:46:02.880030 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852815    5619 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 05:46:02.880030 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852821    5619 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 05:46:02.880030 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852825    5619 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 05:46:02.880030 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852829    5619 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 05:46:02.880030 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852835    5619 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 05:46:02.880030 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852843    5619 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 05:46:02.880030 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852847    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 05:46:02.880030 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852852    5619 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 05:46:02.880030 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852856    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 05:46:02.880030 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852861    5619 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 05:46:02.880030 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852865    5619 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 05:46:02.880776 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852870    5619 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 05:46:02.880776 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852874    5619 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 05:46:02.880776 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852879    5619 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 05:46:02.880776 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852883    5619 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 05:46:02.880776 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852886    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 05:46:02.880776 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852889    5619 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 05:46:02.880776 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852891    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 05:46:02.880776 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852894    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 05:46:02.880776 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852897    5619 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 05:46:02.880776 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852900    5619 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 05:46:02.880776 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852902    5619 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 05:46:02.880776 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852905    5619 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 05:46:02.880776 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852908    5619 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 05:46:02.880776 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852910    5619 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 05:46:02.880776 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852913    5619 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 05:46:02.880776 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852917    5619 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 05:46:02.880776 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852920    5619 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 05:46:02.880776 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852923    5619 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 05:46:02.880776 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852925    5619 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 05:46:02.880776 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852928    5619 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 05:46:02.881608 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852932    5619 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 05:46:02.881608 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852937    5619 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 05:46:02.881608 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852941    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 05:46:02.881608 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852945    5619 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 05:46:02.881608 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852949    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 05:46:02.881608 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852953    5619 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 05:46:02.881608 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852961    5619 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 05:46:02.881608 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852965    5619 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 05:46:02.881608 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852969    5619 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 05:46:02.881608 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852974    5619 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 05:46:02.881608 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852979    5619 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 05:46:02.881608 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852983    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 05:46:02.881608 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852987    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 05:46:02.881608 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852992    5619 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 05:46:02.881608 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.852996    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 05:46:02.881608 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.853001    5619 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 05:46:02.881608 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.853006    5619 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 05:46:02.881608 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.853011    5619 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 05:46:02.881608 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.853016    5619 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 05:46:02.881608 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.853021    5619 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 05:46:02.882572 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.853026    5619 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 05:46:02.882572 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.853030    5619 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 05:46:02.882572 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.853033    5619 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 05:46:02.882572 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.853035    5619 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 05:46:02.882572 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.853038    5619 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 05:46:02.882572 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.853041    5619 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 05:46:02.882572 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:02.853043    5619 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 05:46:02.882572 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.853104    5619 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 05:46:02.882572 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.854396    5619 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 01 05:46:02.882572 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.854457    5619 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 01 05:46:02.882572 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.854709    5619 server.go:1014] "Starting client certificate rotation"
Nov 01 05:46:02.882572 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.854853    5619 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 05:46:02.882572 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.854905    5619 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 05:39:48 +0000 UTC" deadline="2028-03-31 23:42:10.554352744 +0000 UTC"
Nov 01 05:46:02.882572 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.854942    5619 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12401h56m7.699413838s"
Nov 01 05:46:02.882572 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.857529    5619 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 05:46:02.882572 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.859007    5619 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 05:46:02.883272 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.860709    5619 log.go:25] "Validated CRI v1 runtime API"
Nov 01 05:46:02.883272 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.863017    5619 log.go:25] "Validated CRI v1 image API"
Nov 01 05:46:02.883272 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.863886    5619 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 05:46:02.883272 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.867129    5619 fs.go:135] Filesystem UUIDs: map[47b6056d-103c-4176-b5e0-12d2c1a2326b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9c6745f8-50ec-4200-9c72-eea0bd7cf50a:/dev/nvme0n1p3]
Nov 01 05:46:02.883570 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.867163    5619 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15707454db3c61e3767aa29b3d5d424b86a1a1094ac8904dc79e0760c4f332bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15707454db3c61e3767aa29b3d5d424b86a1a1094ac8904dc79e0760c4f332bb/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3549a661c8a27497e5587c136c0879c0b79c8774f773f10c552b8e1865c6cbba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3549a661c8a27497e5587c136c0879c0b79c8774f773f10c552b8e1865c6cbba/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/35a484ca0d3ce1bf1d8f66d1bba9a8a4220a5664e20143504c690bb7277f7642/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/35a484ca0d3ce1bf1d8f66d1bba9a8a4220a5664e20143504c690bb7277f7642/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a75221ba0ae447b96836b7fb9ec29688184b2db03d98b9648de41335a8a6c75/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a75221ba0ae447b96836b7fb9ec29688184b2db03d98b9648de41335a8a6c75/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c168b1451e9ae9c0b21da52fb9636545bbfa9456ed1dc4d957a7aba32dfd671/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c168b1451e9ae9c0b21da52fb9636545bbfa9456ed1dc4d957a7aba32dfd671/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41eba9dd5e22a6c3b1e9cb7102d76937242618bd0230e92edda9fa423e5a6622/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41eba9dd5e22a6c3b1e9cb7102d76937242618bd0230e92edda9fa423e5a6622/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4bab4c0a43a5777b39b5767479c6853f715e7c24e8353ac3e500686a43aef09f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4bab4c0a43a5777b39b5767479c6853f715e7c24e8353ac3e500686a43aef09f/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f7a80cdd4eb9a036dcb5c7db83b97cf9a20d11260749a6dc93fe3d1e61f12f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f7a80cdd4eb9a036dcb5c7db83b97cf9a20d11260749a6dc93fe3d1e61f12f4/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/647768154d03c92133235dfac5255efb626578953a1577b6b4de07a02eb98842/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/647768154d03c92133235dfac5255efb626578953a1577b6b4de07a02eb98842/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/655999dd3abb0a6ce452d86b95ba89143bf9f1914cb7097198ad4fc28b5012de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/655999dd3abb0a6ce452d86b95ba89143bf9f1914cb7097198ad4fc28b5012de/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a64fa9dac836f74899877d2701dccd8af6f843a2ca1367131f63ce41196841a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a64fa9dac836f74899877d2701dccd8af6f843a2ca1367131f63ce41196841a/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b5955782ac24a8a91697cb38fe4f8d59410ba45199f602bbb9aceab6817cc11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b5955782ac24a8a91697cb38fe4f8d59410ba45199f602bbb9aceab6817cc11/userdata/shm major:0 minor:319 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7958929b69dae8fb2c16965e0e68ba60ae49885e0f272cfcb1c28525afcf89e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7958929b69dae8fb2c16965e0e68ba60ae49885e0f272cfcb1c28525afcf89e5/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7c35ac7cdc12d77ae99b72c86cbbf2de2ea1de39beb5e86d7b6c5452c71db176/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7c35ac7cdc12d77ae99b72c86cbbf2de2ea1de39beb5e86d7b6c5452c71db176/userdata/shm major:0 minor:317 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b45d9a43bd1cea46768ba85a72d27e5c6ee0b34f3b7d1db813b95d75a74c225/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b45d9a43bd1cea46768ba85a72d27e5c6ee0b34f3b7d1db813b95d75a74c225/userdata/shm major:0 minor:409 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9caa216e0fa1aa9115b11851c57c842825c6af055b8bb0e5ec35a0619d90d579/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9caa216e0fa1aa9115b11851c57c842825c6af055b8bb0e5ec35a0619d90d579/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af3164d5e42d1871cc9e2571d7ed0458fc9ce0c6b2f407cb740e58d571f0d652/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af3164d5e42d1871cc9e2571d7ed0458fc9ce0c6b2f407cb740e58d571f0d652/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0330c552dede6cbfd8eca1d6ca5c65eaa9a470af6a0338c93d30f2cf853f350/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0330c552dede6cbfd8eca1d6ca5c65eaa9a470af6a0338c93d30f2cf853f350/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b597aaced71271e5cb4b1bbeb4779b21232392778b84a3281a7f298efb137989/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b597aaced71271e5cb4b1bbeb4779b21232392778b84a3281a7f298efb137989/userdata/shm major:0 minor:335 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b61f46a1b913e6c2c2a59d9b2fc4c61b6d9cb47d64d31e5b21f08b4105d3a218/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b61f46a1b913e6c2c2a59d9b2fc4c61b6d9cb47d64d31e5b21f08b4105d3a218/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be6f8f45ab71c357cb49a252f4c1de95650637b9c6931939584d2b2215c10cf3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be6f8f45ab71c357cb49a252f4c1de95650637b9c6931939584d2b2215c10cf3/userdata/shm major:0 minor:315 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc11cb67ba92c1e2a7c27ebe0b6e77b34723fa8217835ad395a71fb8fe3eee8a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc11cb67ba92c1e2a7c27ebe0b6e77b34723fa8217835ad395a71fb8fe3eee8a/userdata/shm major:0 minor:304 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e50e5ec9b15f85b6f2aa2c30ce0d639b58674bb6be0104c1d40b5dacdc10b335/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e50e5ec9b15f85b6f2aa2c30ce0d639b58674bb6be0104c1d40b5dacdc10b335/userdata/shm major:0 minor:333 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed91fd3a410fe1b2ee686b94ba8a4c15a4d1f1872bc6366d6a926bb5c3b1cdc5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed91fd3a410fe1b2ee686b94ba8a4c15a4d1f1872bc6366d6a926bb5c3b1cdc5/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04241194-2fed-4174-9eda-e384dcfd4131/volumes/kubernetes.io~projected/kube-api-access-nxjd2:{mountpoint:/var/lib/kubelet/pods/04241194-2fed-4174-9eda-e384dcfd4131/volumes/kubernetes.io~projected/kube-api-access-nxjd2 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05e09ea1-324b-49ef-a4a5-2645b50232a4/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/05e09ea1-324b-49ef-a4a5-2645b50232a4/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06f0eaf3-04d5-492f-9d3a-6b337091fbea/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/06f0eaf3-04d5-492f-9d3a-6b337091fbea/volumes/kubernetes.io~secret/tls-certificates major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e9a8c0a-32d6-44ef-8a47-f65f6560193d/volumes/kubernetes.io~projected/kube-api-access-tsndz:{mountpoint:/var/lib/kubelet/pods/1e9a8c0a-32d6-44ef-8a47-f65f6560193d/volumes/kubernetes.io~projected/kube-api-access-tsndz major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2980ba94-fa03-419a-82d7-86405b6178ac/volumes/kubernetes.io~projected/kube-api-access-vg2zd:{mountpoint:/var/lib/kubelet/pods/2980ba94-fa03-419a-82d7-86405b6178ac/volumes/kubernetes.io~projected/kube-api-access-vg2zd major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3df5a1c7-a784-4bec-81d1-5ee5d36c707c/volumes/kubernetes.io~projected/kube-api-access-svf57:{mountpoint:/var/lib/kubelet/pods/3df5a1c7-a784-4bec-81d1-5ee5d36c707c/volumes/kubernetes.io~projected/kube-api-access-svf57 major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:398 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~empty-dir/config-out major:0 minor:393 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~projected/kube-api-access-rp4hf:{mountpoint:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~projected/kube-api-access-rp4hf major:0 minor:401 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~projected/tls-assets major:0 minor:399 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/config major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:391 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:395 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:394 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:392 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:387 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/web-config major:0 minor:398 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f767e63-7cc7-4b5e-9941-860fe5f4cd62/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/7f767e63-7cc7-4b5e-9941-860fe5f4cd62/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923/volumes/kubernetes.io~projected/kube-api-access-2hhd5:{mountpoint:/var/lib/kubelet/pods/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923/volumes/kubernetes.io~projected/kube-api-access-2hhd5 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/941ae7ae-e26d-4adb-8d74-729e4c038861/volumes/kubernetes.io~projected/kube-api-access-stntj:{mountpoint:/var/lib/kubelet/pods/941ae7ae-e26d-4adb-8d74-729e4c038861/volumes/kubernetes.io~projected/kube-api-access-stntj major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/941ae7ae-e26d-4adb-8d74-729e4c038861/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/941ae7ae-e26d-4adb-8d74-729e4c038861/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95fd5853-f502-4076-90a2-19fb99f52e2c/volumes/kubernetes.io~projected/kube-api-access-vhq5k:{mountpoint:/var/lib/kubelet/pods/95fd5853-f502-4076-90a2-19fb99f52e2c/volumes/kubernetes.io~projected/kube-api-access-vhq5k major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95fd5853-f502-4076-90a2-19fb99f52e2c/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/95fd5853-f502-4076-90a2-19fb99f52e2c/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95fd5853-f502-4076-90a2-19fb99f52e2c/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/95fd5853-f502-4076-90a2-19fb99f52e2c/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95fd5853-f502-4076-90a2-19fb99f52e2c/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/95fd5853-f502-4076-90a2-19fb99f52e2c/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9efa148c-f7c3-4f9d-bee3-cc1eb353b808/volumes/kubernetes.io~projected/kube-api-access-rprhl:{mountpoint:/var/lib/kubelet/pods/9efa148c-f7c3-4f9d-bee3-cc1eb353b808/volumes/kubernetes.io~projected/kube-api-access-rprhl major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a18b7694-133e-4663-b8f6-a36f981ab717/volumes/kubernetes.io~projected/kube-api-access-kwg57:{mountpoint:/var/lib/kubelet/pods/a18b7694-133e-4663-b8f6-a36f981ab717/volumes/kubernetes.io~projected/kube-api-access-kwg57 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a18b7694-133e-4663-b8f6-a36f981ab717/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/a18b7694-133e-4663-b8f6-a36f981ab717/volumes/kubernetes.io~secret/metrics-certs major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a78d89dc-2ed6-4a2e-9558-f54b73a806d2/volumes/kubernetes.io~projected/kube-api-access-58hcx:{mountpoint:/var/lib/kubelet/pods/a78d89dc-2ed6-4a2e-9558-f54b73a806d2/volumes/kubernetes.io~projected/kube-api-access-58hcx major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa8083b6-e023-46fe-bdb3-a9e99079eb3d/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/aa8083b6-e023-46fe-bdb3-a9e99079eb3d/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa8083b6-e023-46fe-bdb3-a9e99079eb3d/volumes/kubernetes.io~projected/kube-api-access-wk4fp:{mountpoint:/var/lib/kubelet/pods/aa8083b6-e023-46fe-bdb3-a9e99079eb3d/volumes/kubernetes.io~projected/kube-api-access-wk4fp major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa8083b6-e023-46fe-bdb3-a9e99079eb3d/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/aa8083b6-e023-46fe-bdb3-a9e99079eb3d/volumes/kubernetes.io~projected/registry-tls major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa8083b6-e023-46fe-bdb3-a9e99079eb3d/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/aa8083b6-e023-46fe-bdb3-a9e99079eb3d/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa8083b6-e023-46fe-bdb3-a9e99079eb3d/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/aa8083b6-e023-46fe-bdb3-a9e99079eb3d/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bdeea076-cd20-459e-b1a5-79ec825e95e3/volumes/kubernetes.io~projected/kube-api-access-nwxk7:{mountpoint:/var/lib/kubelet/pods/bdeea076-cd20-459e-b1a5-79ec825e95e3/volumes/kubernetes.io~projected/kube-api-access-nwxk7 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1d9c47e-944e-4112-9889-b01892353659/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c1d9c47e-944e-4112-9889-b01892353659/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1d9c47e-944e-4112-9889-b01892353659/volumes/kubernetes.io~projected/kube-api-access-c79lv:{mountpoint:/var/lib/kubelet/pods/c1d9c47e-944e-4112-9889-b01892353659/volumes/kubernetes.io~projected/kube-api-access-c79lv major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1d9c47e-944e-4112-9889-b01892353659/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c1d9c47e-944e-4112-9889-b01892353659/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0e384e0-dcf6-410c-b344-8d797c57e1c9/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d0e384e0-dcf6-410c-b344-8d797c57e1c9/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0e384e0-dcf6-410c-b344-8d797c57e1c9/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d0e384e0-dcf6-410c-b344-8d797c57e1c9/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0e384e0-dcf6-410c-b344-8d797c57e1c9/volumes/kubernetes.io~projected/kube-api-access-86v89:{mountpoint:/var/lib/kubelet/pods/d0e384e0-dcf6-410c-b344-8d797c57e1c9/volumes/kubernetes.io~projected/kube-api-access-86v89 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d937ef17-8f48-475a-955a-e0599c40d25c/volumes/kubernetes.io~projected/kube-api-access-zlqr8:{mountpoint:/var/lib/kubelet/pods/d937ef17-8f48-475a-955a-e0599c40d25c/volumes/kubernetes.io~projected/kube-api-access-zlqr8 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d937ef17-8f48-475a-955a-e0599c40d25c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/d937ef17-8f48-475a-955a-e0599c40d25c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc7de5f7-34ee-46ae-81eb-81717d8f1a52/volumes/kubernetes.io~projected/kube-api-access-w7ttp:{mountpoint:/var/lib/kubelet/pods/dc7de5f7-34ee-46ae-81eb-81717d8f1a52/volumes/kubernetes.io~projected/kube-api-access-w7ttp major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc7de5f7-34ee-46ae-81eb-81717d8f1a52/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/dc7de5f7-34ee-46ae-81eb-81717d8f1a52/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5ace06a-0f90-46ef-8a04-fdc0f7180a1e/volumes/kubernetes.io~projected/kube-api-access-jfpq7:{mountpoint:/var/lib/kubelet/pods/e5ace06a-0f90-46ef-8a04-fdc0f7180a1e/volumes/kubernetes.io~projected/kube-api-access-jfpq7 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8/volumes/kubernetes.io~projected/kube-api-access-zlj5f:{mountpoint:/var/lib/kubelet/pods/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8/volumes/kubernetes.io~projected/kube-api-access-zlj5f major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volume-subpaths/web-config/alertmanager/9 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volume-subpaths/web-config/config-reloader/6 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~empty-dir/config-out major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~projected/kube-api-access-bkn77:{mountpoint:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~projected/kube-api-access-bkn77 major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~projected/tls-assets major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~secret/config-volume major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~secret/web-config major:0 minor:290 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/0b398a9b73df827ec22b571d6cf7ffb43f315c6d280520041f74d8d7aaed692c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/af78ec64e3f798f76760b1197a9495b715145146ecbe04400d1f952391f535ca/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5f1aed0bce9b42501d70898348ee3c7c30739c09f2942f37e3ba93c7bf6c3aee/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/63287530043a0beacedb0b334f9ff377e2d9c9b7f68800b742dc23781b5f0e47/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/dc6d3c44555577e88a6c76553a8b27dcf6542121bdd71fe3e80d929a7f4d49c3/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ed349ebf1b2af3f1c7a775dec57f18047cabd6eaf9f7712b026200a002e94f33/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f5daa5ea7991ee1b14ba4f1f3c623af44b423579b42ac6d1cd9425509fc91668/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/6a4de4ae2d7d3374ad023247ca82975a9c4fa25e38d953263a9ab80128f42580/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/ed5cae655dda6699108a1fd1ac1aa8d558dbc0826947e928e7e3c4b883550880/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/f7fc12215d0f3a27c778bea45defe5e38384a1d70d1d11c18fd97978d70a2ce1/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/aacf10174637efe6acec7223d991806d1b9be460d2a52ce9f41d459eed6e41b4/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/495ed4d85e2046abc7874f0aaafcc84dd863390fc5b4f69405fe076ef6557d09/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/372d37e13852a76fa4349c1fc2f751bcd55d1a50012a3f75e61731b782019782/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/cba0dc06fad8ca3284425eadc20214bb4d971a0281cafbc14faabec460c035bd/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/6d29cd303319e28e48d2b863b86dc07bef67cf178d7b08c1181f1138bdad248d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/307f2898bcbfa1cc8d71a385eb177f4dc29f9c0c438ff62121a8f270b7e35f00/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/52f6e93ee0dbb2188280aec57a761a7dfbb4f4d53474354a5cb22b97b0af4137/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/703fbd1bf76212cd2dd6f3417cbb88d2f7905677b4523fcf8bd724a63d79765c/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/2ba7587c19e97d9ffb0a7b85904bf6d39e75f64b27506c3984d99b8dc3ddff86/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d688ff43b3503078c3a79917686c1e8a800c98b80c473cadf51e5773b0676bd4/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/72d6692c6df0bc3661ca533b8721596dc4fba07069212a3fd56f00e73cb3957a/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/a376b5e1a0e16820b00c5f2336e89e0ff8ea246f071f851513062332f8b80620/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/2c58afa1d9befac83c2ccf43f08bac66671610af6c2040da3feb88802081f963/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/47b0b526002c01e65974e66e5337df7763a6bbcaeac369f1400bde906e251041/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/4f01c6dd6523ea51632af4e5aa2fa2075f1a2170c4e4b8114041ac6b94dd77fc/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/8b9b9eb50ffc3a164bad03e6e7e35f8bc783f771010d6d3995ec8ab622930efc/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/4998b8fc8c4130c9e9c42af7af22aa5cc42c00080ff85fda9d8f0149e1991147/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/55e532467e4d04ba9b0ec4e5cf2e08d578a7de2228bd0728e83d6b852f2eede8/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/9ec15d6e3f3a46661667e7c9d9620b146b0249766149694a7c0c7ea8974be499/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/47e41a2061d39f2ff28eb85e685076da1df5b3d9ded0656d644d2945b3588a38/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/9b9893bf5e241b3e376b0c80a85f06b870e9e1f50fc3182ac750e485ce743a34/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/491bc857787aecde46c33f3cf9491ac6e3dfbc68ff00594d53d1bf3a3c594920/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/74b11be83e1337d16f86ccf902f7d2965e38764b2e2244a3c06885596c31a8d9/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/5f648402fbf76f4821cb201aeaf3bb3426550de9d088e93a12f2e94ac424f715/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/7b530921d54bb9f92f592b11ef7b6685f6ebd56ae5a3ee82c2fc41783d3ad97c/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/9959f4405703f4f22e1ded4195204792c3242cf970873d531c5aacefd91e90ee/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/4f412219b36d6d7b4df9f204e0710c76592f7ea3259ba8c017ec011af57ec98b/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/720870b28ace9e5dce536a62d0cf22727e407ae02e494407ce26e9a8ea22e99c/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/46af7f5e809743d9ebefde8f33d864649c794d02cf4583506ec0d25fc4083280/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/b43db2946406e586e4033e2ff0b752e858a7987a4221843319c26284c12ec1c0/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/dc393cb85615777ec0a41d7b896420ab178d3bd87953e0b92dc63f7d57d64e43/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/e93e82d349cca206ae4900025cb32cf76cd88140a9ca4e75bc9984c82e36bdc5/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/166b440989e4f715258aafc833d3b9b49aec6cc22a87952dd6145f7108581980/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/5ce914d99b87945b47000a2b5b98aab6a962aee8ef75fe6b21796454a21222d3/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/46444d1853ef619567cf022079ab227a347af6000191bcae7f6fdb6d363a5ec4/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/1cceb5010a5adb762e778c321ffd3139b0f5393d5947210cea32b119e2b7ea12/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/fd9de211d62b79d9bdeb88b2a194394d02110c4d73e986280b725e016c7a067d/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/56b91021a9855f6f267fd824d2c60514141a5bbea56f4c94d393d7c502cc8100/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/107ce7abf54149dfcde14913b4159a3b9105e5c4a6ce4af8247fceab23859056/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/88dd5c152198c961ae567d61815b506bb033ea491d450c041c66d19f1dd29760/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/7981b9c5620701ca472a17d23b440880ff325f3bcb9f38428199a7224a185784/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/64078374f3c6ceb994e7c39e13c180f480e305d6d5f41bbf4e3305bfc7db82af/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/e1060630008bc0ed48e96e27684a86d4da231a1fc5af4a9a199e8960b2a037cc/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/4f7c2e7a2db1a224556bdf68e8c86741a996a470bb4a848668b9cc42f6e75d69/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/8ef4c76d6e1841e547989dde092ed187af4a425762589cf9ddaab2d6cee3fb98/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/a60bafead055d630052d4f23251bede76a69418914688b7aaccf86839fc788ba/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/b211e50d0c98a603f5d50e0816195be211cf074ca70bc1778eda983853e7fdc2/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/18821a4f4d07c08510cfaa1cf713e3f51655fd57ff0c63b072f91b09a7a10eb5/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/56f1363831030066a6402cf7733c894989f5a075f64ab9820f094edcaef5b01c/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/8ac35556ca11af65222cc0f00fc07e63b4cd6acf11da34de494af1ba8de764b6/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/1c13814ad2d64fc4bf9b362d0fd3f97b352e15d2ed197cdf62eab489d47b1180/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/3ae4de6ab80512bac9009bf6ab113208db1adfa23c04d1bb199341a82bd42da9/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/4be8718c0387da41d811e868bf2b362654125eda425d58159e895fcc767bb2e0/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/5600b9a490c9d2827e5b1abf63142a7d8fba194c5f132c4159c30080187dc40f/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/c0cab2de07875feebe795448bbba188457d785f91f11ea30c63f8c9bfc982491/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/b72cb1e0ca186a1a56314e0a7389b051aabe3cdab795b0e14ec2001be5239d07/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/89fad278734698333d416f7d778f7d6c06a5a79c33b2734188970d87ffc5d409/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/95b7648113446d32c39db71ddf901a79cc1b7a753901640f1ddd2dd751d801e8/merged major:0 minor:97 fsType:overlay blockSize:0}]
Nov 01 05:46:02.883988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.879281    5619 manager.go:217] Machine: {Timestamp:2026-11-01 05:46:02.878459838 +0000 UTC m=+0.089283341 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099962 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c281b553c640ecb4e199644f9666b SystemUUID:ec2c281b-553c-640e-cb4e-199644f9666b BootID:5c4d2b0c-4260-4bf2-8720-61ca0aaf76de Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f7a80cdd4eb9a036dcb5c7db83b97cf9a20d11260749a6dc93fe3d1e61f12f4/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b45d9a43bd1cea46768ba85a72d27e5c6ee0b34f3b7d1db813b95d75a74c225/userdata/shm DeviceMajor:0 DeviceMinor:409 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af3164d5e42d1871cc9e2571d7ed0458fc9ce0c6b2f407cb740e58d571f0d652/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:272 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:396 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1e9a8c0a-32d6-44ef-8a47-f65f6560193d/volumes/kubernetes.io~projected/kube-api-access-tsndz DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7958929b69dae8fb2c16965e0e68ba60ae49885e0f272cfcb1c28525afcf89e5/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:298 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15707454db3c61e3767aa29b3d5d424b86a1a1094ac8904dc79e0760c4f332bb/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4bab4c0a43a5777b39b5767479c6853f715e7c24e8353ac3e500686a43aef09f/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95fd5853-f502-4076-90a2-19fb99f52e2c/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:331 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~projected/kube-api-access-rp4hf DeviceMajor:0 DeviceMinor:401 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3549a661c8a27497e5587c136c0879c0b79c8774f773f10c552b8e1865c6cbba/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aa8083b6-e023-46fe-bdb3-a9e99079eb3d/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:395 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c1d9c47e-944e-4112-9889-b01892353659/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d0e384e0-dcf6-410c-b344-8d797c57e1c9/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b61f46a1b913e6c2c2a59d9b2fc4c61b6d9cb47d64d31e5b21f08b4105d3a218/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:294 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/655999dd3abb0a6ce452d86b95ba89143bf9f1914cb7097198ad4fc28b5012de/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a18b7694-133e-4663-b8f6-a36f981ab717/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7c35ac7cdc12d77ae99b72c86cbbf2de2ea1de39beb5e86d7b6c5452c71db176/userdata/shm DeviceMajor:0 DeviceMinor:317 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:398 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/05e09ea1-324b-49ef-a4a5-2645b50232a4/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc11cb67ba92c1e2a7c27ebe0b6e77b34723fa8217835ad395a71fb8fe3eee8a/userdata/shm DeviceMajor:0 DeviceMinor:304 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bdeea076-cd20-459e-b1a5-79ec825e95e3/volumes/kubernetes.io~projected/kube-api-access-nwxk7 DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:297 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b597aaced71271e5cb4b1bbeb4779b21232392778b84a3281a7f298efb137989/userdata/shm DeviceMajor:0 DeviceMinor:335 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:387 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:392 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed91fd3a410fe1b2ee686b94ba8a4c15a4d1f1872bc6366d6a926bb5c3b1cdc5/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:296 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1d9c47e-944e-4112-9889-b01892353659/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:282 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be6f8f45ab71c357cb49a252f4c1de95650637b9c6931939584d2b2215c10cf3/userdata/shm DeviceMajor:0 DeviceMinor:315 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:394 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41eba9dd5e22a6c3b1e9cb7102d76937242618bd0230e92edda9fa423e5a6622/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa8083b6-e023-46fe-bdb3-a9e99079eb3d/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aa8083b6-e023-46fe-bdb3-a9e99079eb3d/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:399 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a78d89dc-2ed6-4a2e-9558-f54b73a806d2/volumes/kubernetes.io~projected/kube-api-access-58hcx DeviceMajor:0 DeviceMinor:314 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/941ae7ae-e26d-4adb-8d74-729e4c038861/volumes/kubernetes.io~projected/kube-api-access-stntj DeviceMajor:0 DeviceMinor:326 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a18b7694-133e-4663-b8f6-a36f981ab717/volumes/kubernetes.io~projected/kube-api-access-kwg57 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~projected/kube-api-access-bkn77 DeviceMajor:0 DeviceMinor:305 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:398 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:391 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9caa216e0fa1aa9115b11851c57c842825c6af055b8bb0e5ec35a0619d90d579/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dc7de5f7-34ee-46ae-81eb-81717d8f1a52/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aa8083b6-e023-46fe-bdb3-a9e99079eb3d/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95fd5853-f502-4076-90a2-19fb99f52e2c/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:327 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/95fd5853-f502-4076-90a2-19fb99f52e2c/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:330 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d937ef17-8f48-475a-955a-e0599c40d25c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3df5a1c7-a784-4bec-81d1-5ee5d36c707c/volumes/kubernetes.io~projected/kube-api-access-svf57 DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:301 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:397 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/35a484ca0d3ce1bf1d8f66d1bba9a8a4220a5664e20143504c690bb7277f7642/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0e384e0-dcf6-410c-b344-8d797c57e1c9/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e50e5ec9b15f85b6f2aa2c30ce0d639b58674bb6be0104c1d40b5dacdc10b335/userdata/shm DeviceMajor:0 DeviceMinor:333 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:400 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e5ace06a-0f90-46ef-8a04-fdc0f7180a1e/volumes/kubernetes.io~projected/kube-api-access-jfpq7 DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0330c552dede6cbfd8eca1d6ca5c65eaa9a470af6a0338c93d30f2cf853f350/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:300 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aa8083b6-e023-46fe-bdb3-a9e99079eb3d/volumes/kubernetes.io~projected/kube-api-access-wk4fp DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b5955782ac24a8a91697cb38fe4f8d59410ba45199f602bbb9aceab6817cc11/userdata/shm DeviceMajor:0 DeviceMinor:319 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2980ba94-fa03-419a-82d7-86405b6178ac/volumes/kubernetes.io~projected/kube-api-access-vg2zd DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d937ef17-8f48-475a-955a-e0599c40d25c/volumes/kubernetes.io~projected/kube-api-access-zlqr8 DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:393 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0e384e0-dcf6-410c-b344-8d797c57e1c9/volumes/kubernetes.io~projected/kube-api-access-86v89 DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/04241194-2fed-4174-9eda-e384dcfd4131/volumes/kubernetes.io~projected/kube-api-access-nxjd2 DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a64fa9dac836f74899877d2701dccd8af6f843a2ca1367131f63ce41196841a/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/06f0eaf3-04d5-492f-9d3a-6b337091fbea/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dc7de5f7-34ee-46ae-81eb-81717d8f1a52/volumes/kubernetes.io~projected/kube-api-access-w7ttp DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8/volumes/kubernetes.io~projected/kube-api-access-zlj5f DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9efa148c-f7c3-4f9d-bee3-cc1eb353b808/volumes/kubernetes.io~projected/kube-api-access-rprhl DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a75221ba0ae447b96836b7fb9ec29688184b2db03d98b9648de41335a8a6c75/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923/volumes/kubernetes.io~projected/kube-api-access-2hhd5 DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/647768154d03c92133235dfac5255efb626578953a1577b6b4de07a02eb98842/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/941ae7ae-e26d-4adb-8d74-729e4c038861/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:325 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/95fd5853-f502-4076-90a2-19fb99f52e2c/volumes/kubernetes.io~projected/kube-api-access-vhq5k DeviceMajor:0 DeviceMinor:332 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c168b1451e9ae9c0b21da52fb9636545bbfa9456ed1dc4d957a7aba32dfd671/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c1d9c47e-944e-4112-9889-b01892353659/volumes/kubernetes.io~projected/kube-api-access-c79lv DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f767e63-7cc7-4b5e-9941-860fe5f4cd62/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:313 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:3c168b1451e9ae9 MacAddress:d2:34:89:dd:b1:9a Speed:10000 Mtu:8901} {Name:647768154d03c92 MacAddress:2a:17:24:6b:d2:fc Speed:10000 Mtu:8901} {Name:655999dd3abb0a6 MacAddress:de:5b:cd:6b:3f:92 Speed:10000 Mtu:8901} {Name:6b5955782ac24a8 MacAddress:9e:33:8e:cf:b7:13 Speed:10000 Mtu:8901} {Name:7958929b69dae8f MacAddress:16:68:0a:33:b2:39 Speed:10000 Mtu:8901} {Name:7c35ac7cdc12d77 MacAddress:12:5e:51:0d:ca:71 Speed:10000 Mtu:8901} {Name:8b45d9a43bd1cea MacAddress:ba:38:68:ff:f1:25 Speed:10000 Mtu:8901} {Name:b597aaced71271e MacAddress:a6:ad:d2:46:d4:4f Speed:10000 Mtu:8901} {Name:b61f46a1b913e6c MacAddress:b6:56:a9:a2:5d:f7 Speed:10000 Mtu:8901} {Name:be6f8f45ab71c35 MacAddress:b6:94:7d:9c:d6:41 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:4f:20:67:74:0b Speed:0 Mtu:9001} {Name:br-int MacAddress:32:ed:4a:be:06:04 Speed:0 Mtu:8901} {Name:cc11cb67ba92c1e MacAddress:9e:b2:7e:d5:8f:f4 Speed:10000 Mtu:8901} {Name:e50e5ec9b15f85b MacAddress:0a:f0:8a:01:50:91 Speed:10000 Mtu:8901} {Name:ed91fd3a410fe1b MacAddress:e2:55:75:a5:14:43 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:4f:20:67:74:0b Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:2a:4c:c8:3f:16:b0 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:d6:58:05:4a:01:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 05:46:02.883988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.880037    5619 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 05:46:02.883988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.880105    5619 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 05:46:02.883988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.882842    5619 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 05:46:02.883988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.882889    5619 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-213.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 05:46:02.883988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.883098    5619 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 05:46:02.883988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.883112    5619 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 05:46:02.883988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.883128    5619 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 05:46:02.883988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.883157    5619 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 05:46:02.883988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.883306    5619 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 05:46:02.883988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.883567    5619 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 05:46:02.883988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.883713    5619 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 05:46:02.883988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.883760    5619 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 05:46:02.883988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.883780    5619 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 05:46:02.883988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.883794    5619 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 05:46:02.883988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.883809    5619 kubelet.go:405] "Adding apiserver pod source"
Nov 01 05:46:02.883988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.883823    5619 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 05:46:02.885012 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.884831    5619 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 05:46:02.885012 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.884846    5619 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 05:46:02.885548 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.885526    5619 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 05:46:02.902748 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.902724    5619 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 05:46:02.902748 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.902745    5619 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 05:46:02.902975 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.902903    5619 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 01 05:46:02.903169 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.903153    5619 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 05:46:02.903250 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.903193    5619 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 05:46:02.903362 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.903344    5619 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 05:46:02.903362 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.903365    5619 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 05:46:02.903362 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.903377    5619 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 05:46:02.903548 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.903386    5619 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 05:46:02.903548 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.903392    5619 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 05:46:02.903548 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.903401    5619 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 05:46:02.903548 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.903409    5619 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 05:46:02.903548 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.903415    5619 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 05:46:02.903548 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.903435    5619 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 05:46:02.903548 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.903444    5619 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 05:46:02.903548 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.903466    5619 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 05:46:02.903548 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.903476    5619 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 05:46:02.903548 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.903507    5619 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 05:46:02.903548 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.903516    5619 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 05:46:02.907484 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.907452    5619 server.go:1263] "Started kubelet"
Nov 01 05:46:02.907825 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.907595    5619 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 05:46:02.907998 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.907841    5619 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 05:46:02.907998 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.907933    5619 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 05:46:02.907993 ip-10-0-129-213 systemd[1]: Started Kubernetes Kubelet.
Nov 01 05:46:02.909138 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.909023    5619 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 05:46:02.911289 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.909708    5619 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 05:46:02.918773 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.913037    5619 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 05:46:02.918773 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.913468    5619 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 05:46:02.918773 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:02.913631    5619 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-213.us-west-1.compute.internal\" not found"
Nov 01 05:46:02.918773 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.913685    5619 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 05:46:02.918773 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.913697    5619 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 05:46:02.918773 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.913930    5619 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 05:39:49 +0000 UTC" deadline="2028-05-23 09:58:37.699630863 +0000 UTC"
Nov 01 05:46:02.918773 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.913983    5619 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13660h12m34.785650721s"
Nov 01 05:46:02.918773 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.914219    5619 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 05:46:02.918773 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.917774    5619 factory.go:55] Registering systemd factory
Nov 01 05:46:02.918773 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.917798    5619 factory.go:223] Registration of the systemd container factory successfully
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.921404    5619 factory.go:153] Registering CRI-O factory
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.921428    5619 factory.go:223] Registration of the crio container factory successfully
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.921499    5619 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.921524    5619 factory.go:103] Registering Raw factory
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.921541    5619 manager.go:1199] Started watching for new ooms in manager
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.921896    5619 manager.go:319] Starting recovery of all containers
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922086    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2980ba94-fa03-419a-82d7-86405b6178ac" volumeName="kubernetes.io/configmap/2980ba94-fa03-419a-82d7-86405b6178ac-cni-binary-copy" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922128    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" volumeName="kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922145    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" volumeName="kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-web-config" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922160    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1d9c47e-944e-4112-9889-b01892353659" volumeName="kubernetes.io/secret/c1d9c47e-944e-4112-9889-b01892353659-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922174    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d937ef17-8f48-475a-955a-e0599c40d25c" volumeName="kubernetes.io/secret/d937ef17-8f48-475a-955a-e0599c40d25c-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922187    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e9a8c0a-32d6-44ef-8a47-f65f6560193d" volumeName="kubernetes.io/projected/1e9a8c0a-32d6-44ef-8a47-f65f6560193d-kube-api-access-tsndz" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922201    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3df5a1c7-a784-4bec-81d1-5ee5d36c707c" volumeName="kubernetes.io/empty-dir/3df5a1c7-a784-4bec-81d1-5ee5d36c707c-tmp" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922214    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" volumeName="kubernetes.io/empty-dir/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-config-out" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922231    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a18b7694-133e-4663-b8f6-a36f981ab717" volumeName="kubernetes.io/projected/a18b7694-133e-4663-b8f6-a36f981ab717-kube-api-access-kwg57" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922244    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdeea076-cd20-459e-b1a5-79ec825e95e3" volumeName="kubernetes.io/configmap/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-daemon-config" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922256    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4eea300-bfd9-465e-9276-16494acaa883" volumeName="kubernetes.io/empty-dir/f4eea300-bfd9-465e-9276-16494acaa883-alertmanager-main-db" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922271    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" volumeName="kubernetes.io/empty-dir/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-db" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922284    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" volumeName="kubernetes.io/projected/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-tls-assets" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922297    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="910a0fc6-8aa8-4b22-b767-b0ecf3b6c923" volumeName="kubernetes.io/configmap/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-config-volume" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922347    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="910a0fc6-8aa8-4b22-b767-b0ecf3b6c923" volumeName="kubernetes.io/secret/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-metrics-tls" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922362    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95fd5853-f502-4076-90a2-19fb99f52e2c" volumeName="kubernetes.io/secret/95fd5853-f502-4076-90a2-19fb99f52e2c-secret-metrics-server-tls" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922374    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa8083b6-e023-46fe-bdb3-a9e99079eb3d" volumeName="kubernetes.io/secret/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-image-registry-private-configuration" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922386    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec642c5b-c364-4faa-94d5-bb5bf32a2cb8" volumeName="kubernetes.io/configmap/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922399    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4eea300-bfd9-465e-9276-16494acaa883" volumeName="kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922412    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa8083b6-e023-46fe-bdb3-a9e99079eb3d" volumeName="kubernetes.io/projected/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-registry-tls" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922425    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d937ef17-8f48-475a-955a-e0599c40d25c" volumeName="kubernetes.io/projected/d937ef17-8f48-475a-955a-e0599c40d25c-kube-api-access-zlqr8" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922436    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4eea300-bfd9-465e-9276-16494acaa883" volumeName="kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-web-config" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922471    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec642c5b-c364-4faa-94d5-bb5bf32a2cb8" volumeName="kubernetes.io/empty-dir/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-textfile" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922485    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec642c5b-c364-4faa-94d5-bb5bf32a2cb8" volumeName="kubernetes.io/secret/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-tls" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922497    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec642c5b-c364-4faa-94d5-bb5bf32a2cb8" volumeName="kubernetes.io/secret/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922513    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4eea300-bfd9-465e-9276-16494acaa883" volumeName="kubernetes.io/configmap/f4eea300-bfd9-465e-9276-16494acaa883-metrics-client-ca" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922527    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4eea300-bfd9-465e-9276-16494acaa883" volumeName="kubernetes.io/projected/f4eea300-bfd9-465e-9276-16494acaa883-kube-api-access-bkn77" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922543    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4eea300-bfd9-465e-9276-16494acaa883" volumeName="kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-cluster-tls-config" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922562    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4eea300-bfd9-465e-9276-16494acaa883" volumeName="kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-config-volume" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922575    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2980ba94-fa03-419a-82d7-86405b6178ac" volumeName="kubernetes.io/configmap/2980ba94-fa03-419a-82d7-86405b6178ac-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922588    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2980ba94-fa03-419a-82d7-86405b6178ac" volumeName="kubernetes.io/projected/2980ba94-fa03-419a-82d7-86405b6178ac-kube-api-access-vg2zd" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922602    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" volumeName="kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-tls" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922620    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" volumeName="kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-config" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922633    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0e384e0-dcf6-410c-b344-8d797c57e1c9" volumeName="kubernetes.io/empty-dir/d0e384e0-dcf6-410c-b344-8d797c57e1c9-tmp" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922648    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc7de5f7-34ee-46ae-81eb-81717d8f1a52" volumeName="kubernetes.io/projected/dc7de5f7-34ee-46ae-81eb-81717d8f1a52-kube-api-access-w7ttp" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922661    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5ace06a-0f90-46ef-8a04-fdc0f7180a1e" volumeName="kubernetes.io/projected/e5ace06a-0f90-46ef-8a04-fdc0f7180a1e-kube-api-access-jfpq7" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922673    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4eea300-bfd9-465e-9276-16494acaa883" volumeName="kubernetes.io/configmap/f4eea300-bfd9-465e-9276-16494acaa883-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922687    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" volumeName="kubernetes.io/projected/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-kube-api-access-rp4hf" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922701    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9efa148c-f7c3-4f9d-bee3-cc1eb353b808" volumeName="kubernetes.io/projected/9efa148c-f7c3-4f9d-bee3-cc1eb353b808-kube-api-access-rprhl" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922716    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec642c5b-c364-4faa-94d5-bb5bf32a2cb8" volumeName="kubernetes.io/projected/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-kube-api-access-zlj5f" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922732    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4eea300-bfd9-465e-9276-16494acaa883" volumeName="kubernetes.io/empty-dir/f4eea300-bfd9-465e-9276-16494acaa883-config-out" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922745    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4eea300-bfd9-465e-9276-16494acaa883" volumeName="kubernetes.io/projected/f4eea300-bfd9-465e-9276-16494acaa883-tls-assets" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922759    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" volumeName="kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-kube-rbac-proxy" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922775    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" volumeName="kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-metrics-client-certs" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922788    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="941ae7ae-e26d-4adb-8d74-729e4c038861" volumeName="kubernetes.io/secret/941ae7ae-e26d-4adb-8d74-729e4c038861-monitoring-plugin-cert" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922801    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa8083b6-e023-46fe-bdb3-a9e99079eb3d" volumeName="kubernetes.io/configmap/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-trusted-ca" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922813    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa8083b6-e023-46fe-bdb3-a9e99079eb3d" volumeName="kubernetes.io/empty-dir/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-ca-trust-extracted" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922826    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa8083b6-e023-46fe-bdb3-a9e99079eb3d" volumeName="kubernetes.io/projected/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-bound-sa-token" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922839    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1d9c47e-944e-4112-9889-b01892353659" volumeName="kubernetes.io/configmap/c1d9c47e-944e-4112-9889-b01892353659-ovnkube-script-lib" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922853    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4eea300-bfd9-465e-9276-16494acaa883" volumeName="kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922867    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="910a0fc6-8aa8-4b22-b767-b0ecf3b6c923" volumeName="kubernetes.io/projected/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-kube-api-access-2hhd5" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922881    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" volumeName="kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922893    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" volumeName="kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922906    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1d9c47e-944e-4112-9889-b01892353659" volumeName="kubernetes.io/projected/c1d9c47e-944e-4112-9889-b01892353659-kube-api-access-c79lv" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922920    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3df5a1c7-a784-4bec-81d1-5ee5d36c707c" volumeName="kubernetes.io/projected/3df5a1c7-a784-4bec-81d1-5ee5d36c707c-kube-api-access-svf57" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922934    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" volumeName="kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922980    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95fd5853-f502-4076-90a2-19fb99f52e2c" volumeName="kubernetes.io/configmap/95fd5853-f502-4076-90a2-19fb99f52e2c-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.922995    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95fd5853-f502-4076-90a2-19fb99f52e2c" volumeName="kubernetes.io/empty-dir/95fd5853-f502-4076-90a2-19fb99f52e2c-audit-log" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923011    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95fd5853-f502-4076-90a2-19fb99f52e2c" volumeName="kubernetes.io/secret/95fd5853-f502-4076-90a2-19fb99f52e2c-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923024    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a18b7694-133e-4663-b8f6-a36f981ab717" volumeName="kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923040    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a78d89dc-2ed6-4a2e-9558-f54b73a806d2" volumeName="kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923055    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04241194-2fed-4174-9eda-e384dcfd4131" volumeName="kubernetes.io/projected/04241194-2fed-4174-9eda-e384dcfd4131-kube-api-access-nxjd2" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923069    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" volumeName="kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923082    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" volumeName="kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-grpc-tls" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923094    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95fd5853-f502-4076-90a2-19fb99f52e2c" volumeName="kubernetes.io/configmap/95fd5853-f502-4076-90a2-19fb99f52e2c-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923107    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9efa148c-f7c3-4f9d-bee3-cc1eb353b808" volumeName="kubernetes.io/configmap/9efa148c-f7c3-4f9d-bee3-cc1eb353b808-iptables-alerter-script" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923119    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa8083b6-e023-46fe-bdb3-a9e99079eb3d" volumeName="kubernetes.io/projected/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-kube-api-access-wk4fp" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923133    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1d9c47e-944e-4112-9889-b01892353659" volumeName="kubernetes.io/configmap/c1d9c47e-944e-4112-9889-b01892353659-ovnkube-config" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923160    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" volumeName="kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923177    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" volumeName="kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923194    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa8083b6-e023-46fe-bdb3-a9e99079eb3d" volumeName="kubernetes.io/configmap/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-registry-certificates" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923207    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdeea076-cd20-459e-b1a5-79ec825e95e3" volumeName="kubernetes.io/configmap/bdeea076-cd20-459e-b1a5-79ec825e95e3-cni-binary-copy" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923220    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d937ef17-8f48-475a-955a-e0599c40d25c" volumeName="kubernetes.io/configmap/d937ef17-8f48-475a-955a-e0599c40d25c-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923233    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" volumeName="kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923247    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f767e63-7cc7-4b5e-9941-860fe5f4cd62" volumeName="kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923261    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0e384e0-dcf6-410c-b344-8d797c57e1c9" volumeName="kubernetes.io/projected/d0e384e0-dcf6-410c-b344-8d797c57e1c9-kube-api-access-86v89" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923275    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4eea300-bfd9-465e-9276-16494acaa883" volumeName="kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923287    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4eea300-bfd9-465e-9276-16494acaa883" volumeName="kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923300    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05e09ea1-324b-49ef-a4a5-2645b50232a4" volumeName="kubernetes.io/configmap/05e09ea1-324b-49ef-a4a5-2645b50232a4-konnectivity-ca" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923315    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" volumeName="kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-thanos-prometheus-http-client-file" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923344    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95fd5853-f502-4076-90a2-19fb99f52e2c" volumeName="kubernetes.io/projected/95fd5853-f502-4076-90a2-19fb99f52e2c-kube-api-access-vhq5k" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923356    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95fd5853-f502-4076-90a2-19fb99f52e2c" volumeName="kubernetes.io/secret/95fd5853-f502-4076-90a2-19fb99f52e2c-client-ca-bundle" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923368    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa8083b6-e023-46fe-bdb3-a9e99079eb3d" volumeName="kubernetes.io/secret/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-installation-pull-secrets" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923381    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d937ef17-8f48-475a-955a-e0599c40d25c" volumeName="kubernetes.io/empty-dir/d937ef17-8f48-475a-955a-e0599c40d25c-data-volume" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923396    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" volumeName="kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-metrics-client-ca" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923409    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="910a0fc6-8aa8-4b22-b767-b0ecf3b6c923" volumeName="kubernetes.io/empty-dir/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-tmp-dir" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923423    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1d9c47e-944e-4112-9889-b01892353659" volumeName="kubernetes.io/configmap/c1d9c47e-944e-4112-9889-b01892353659-env-overrides" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923437    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0e384e0-dcf6-410c-b344-8d797c57e1c9" volumeName="kubernetes.io/empty-dir/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-tuned" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923450    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc7de5f7-34ee-46ae-81eb-81717d8f1a52" volumeName="kubernetes.io/secret/dc7de5f7-34ee-46ae-81eb-81717d8f1a52-cert" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923457    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec642c5b-c364-4faa-94d5-bb5bf32a2cb8" volumeName="kubernetes.io/configmap/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-metrics-client-ca" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923467    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2980ba94-fa03-419a-82d7-86405b6178ac" volumeName="kubernetes.io/configmap/2980ba94-fa03-419a-82d7-86405b6178ac-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923480    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="941ae7ae-e26d-4adb-8d74-729e4c038861" volumeName="kubernetes.io/projected/941ae7ae-e26d-4adb-8d74-729e4c038861-kube-api-access-stntj" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923492    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdeea076-cd20-459e-b1a5-79ec825e95e3" volumeName="kubernetes.io/projected/bdeea076-cd20-459e-b1a5-79ec825e95e3-kube-api-access-nwxk7" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923501    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5ace06a-0f90-46ef-8a04-fdc0f7180a1e" volumeName="kubernetes.io/configmap/e5ace06a-0f90-46ef-8a04-fdc0f7180a1e-serviceca" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923508    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05e09ea1-324b-49ef-a4a5-2645b50232a4" volumeName="kubernetes.io/secret/05e09ea1-324b-49ef-a4a5-2645b50232a4-agent-certs" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923515    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06f0eaf3-04d5-492f-9d3a-6b337091fbea" volumeName="kubernetes.io/secret/06f0eaf3-04d5-492f-9d3a-6b337091fbea-tls-certificates" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923523    5619 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e9a8c0a-32d6-44ef-8a47-f65f6560193d" volumeName="kubernetes.io/empty-dir/1e9a8c0a-32d6-44ef-8a47-f65f6560193d-tmp-dir" seLinuxMountContext=""
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923530    5619 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 05:46:02.930217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.923536    5619 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 05:46:02.952094 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.952054    5619 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 05:46:02.954302 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.953965    5619 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 05:46:02.954302 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.954032    5619 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 05:46:02.954302 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.954112    5619 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 05:46:02.956384 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:02.954314    5619 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 05:46:02.969281 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.969193    5619 generic.go:356] "Generic (PLEG): container finished" podID="ec642c5b-c364-4faa-94d5-bb5bf32a2cb8" containerID="f3af80acba16092e7853bb55c19aa23f499052f2c83ebed5f2f7e29fd5077ce8" exitCode=0
Nov 01 05:46:02.988781 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:02.988750    5619 generic.go:356] "Generic (PLEG): container finished" podID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerID="0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03" exitCode=0
Nov 01 05:46:03.011550 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.011509    5619 generic.go:356] "Generic (PLEG): container finished" podID="28a06cb4c587db3aac34f3d8afd3bbd7" containerID="41139941351a4601d5171e7c8eff0a7feddffde463f7d759201ef7a8f4ec8ce9" exitCode=0
Nov 01 05:46:03.013758 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:03.013730    5619 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-213.us-west-1.compute.internal\" not found"
Nov 01 05:46:03.023633 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.023593    5619 generic.go:356] "Generic (PLEG): container finished" podID="f4eea300-bfd9-465e-9276-16494acaa883" containerID="5f71a589277cb4dd289414e7212d5b8b60008db7afae72656b26c20e0134450a" exitCode=0
Nov 01 05:46:03.048580 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.048553    5619 manager.go:324] Recovery completed
Nov 01 05:46:03.051043 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.051008    5619 generic.go:356] "Generic (PLEG): container finished" podID="2980ba94-fa03-419a-82d7-86405b6178ac" containerID="541e8cdc298f1f321484d562988b6dbac91f943703c3f2b2eea3fcec3d0044b3" exitCode=0
Nov 01 05:46:03.051043 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.051042    5619 generic.go:356] "Generic (PLEG): container finished" podID="2980ba94-fa03-419a-82d7-86405b6178ac" containerID="6f5a7e9fd77d3a850d78b77c143caf23e8653bc60d4fdda10d1ee5df08d5b6c2" exitCode=0
Nov 01 05:46:03.051355 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.051051    5619 generic.go:356] "Generic (PLEG): container finished" podID="2980ba94-fa03-419a-82d7-86405b6178ac" containerID="ae1ddcd938354d5d2782f4e1bd7c14a2ec400c49929176e0b34bae2287359e05" exitCode=0
Nov 01 05:46:03.051355 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.051060    5619 generic.go:356] "Generic (PLEG): container finished" podID="2980ba94-fa03-419a-82d7-86405b6178ac" containerID="d905f4c1a2c65acd1023f316e58c750290a7ceb3e9dc4148c1211c9beb8cd1d7" exitCode=0
Nov 01 05:46:03.051355 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.051066    5619 generic.go:356] "Generic (PLEG): container finished" podID="2980ba94-fa03-419a-82d7-86405b6178ac" containerID="c25f4e50ad87fa964adc6b9538fecde359cf135a451acb50c9d6be1ffddd4078" exitCode=0
Nov 01 05:46:03.051355 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.051074    5619 generic.go:356] "Generic (PLEG): container finished" podID="2980ba94-fa03-419a-82d7-86405b6178ac" containerID="5b61f46fa443e030a233667f724107f0a964e69e71ef468642e66501d8b46bdf" exitCode=0
Nov 01 05:46:03.052392 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.052311    5619 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:46:03.054542 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:03.054510    5619 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 01 05:46:03.074661 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.074630    5619 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:46:03.076699 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.076671    5619 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:46:03.076800 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.076710    5619 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:46:03.076800 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.076724    5619 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:46:03.079876 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.079854    5619 cpu_manager.go:226] "Starting" policy="none"
Nov 01 05:46:03.079876 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.079872    5619 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 05:46:03.080014 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.079883    5619 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:46:03.080014 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.079896    5619 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 05:46:03.080117 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.080060    5619 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 01 05:46:03.080117 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.080075    5619 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 01 05:46:03.080117 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.080099    5619 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 01 05:46:03.080117 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.080107    5619 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 01 05:46:03.080117 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.080114    5619 policy_none.go:50] "Start"
Nov 01 05:46:03.080292 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.080125    5619 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 05:46:03.080292 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.080140    5619 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 05:46:03.080397 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.080302    5619 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 01 05:46:03.080397 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.080313    5619 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 01 05:46:03.080397 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.080349    5619 policy_none.go:44] "Start"
Nov 01 05:46:03.081302 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.081279    5619 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:46:03.081395 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.081331    5619 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:46:03.081395 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.081348    5619 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:46:03.086189 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.086155    5619 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 05:46:03.090690 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.090676    5619 manager.go:341] "Starting Device Plugin manager"
Nov 01 05:46:03.090782 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:03.090702    5619 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 05:46:03.090782 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.090715    5619 server.go:87] "Starting device plugin registration server"
Nov 01 05:46:03.091014 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.090997    5619 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 05:46:03.091718 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.091015    5619 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 05:46:03.091718 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.091667    5619 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 05:46:03.091882 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.091794    5619 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 05:46:03.091882 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.091836    5619 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 05:46:03.091882 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.091843    5619 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 05:46:03.092071 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.091900    5619 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T05:46:03.091822022Z","UUID":"12fcea12-064c-44ca-b266-8b59b3e9dd47","Handler":null,"Name":"","Endpoint":""}
Nov 01 05:46:03.093060 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:03.093035    5619 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 05:46:03.093134 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:03.093089    5619 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-213.us-west-1.compute.internal\" not found"
Nov 01 05:46:03.093641 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.093625    5619 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 05:46:03.093731 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.093650    5619 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 05:46:03.101808 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.101779    5619 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:46:03.103014 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.102993    5619 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:46:03.192509 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.192468    5619 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 05:46:03.193957 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.193931    5619 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 05:46:03.194098 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.193970    5619 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 05:46:03.194098 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.193987    5619 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-213.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 05:46:03.194098 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.194011    5619 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:46:03.203768 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.203744    5619 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 05:46:03.203957 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.203940    5619 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 01 05:46:03.255668 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.255564    5619 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal"]
Nov 01 05:46:03.256396 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.256371    5619 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:46:03.256656 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.256608    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal" event={"ID":"28a06cb4c587db3aac34f3d8afd3bbd7","Type":"ContainerStarted","Data":"109981c288c5c2a2a54b1f820e095bf3c7b1274d1ce9ef2bdd8665903bf1a859"}
Nov 01 05:46:03.256776 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.256663    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal" event={"ID":"28a06cb4c587db3aac34f3d8afd3bbd7","Type":"ContainerDied","Data":"41139941351a4601d5171e7c8eff0a7feddffde463f7d759201ef7a8f4ec8ce9"}
Nov 01 05:46:03.256776 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.256682    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal" event={"ID":"28a06cb4c587db3aac34f3d8afd3bbd7","Type":"ContainerStarted","Data":"6a64fa9dac836f74899877d2701dccd8af6f843a2ca1367131f63ce41196841a"}
Nov 01 05:46:03.256776 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.256699    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal" event={"ID":"ef0f634a7979920e9df606f3bb474aac","Type":"ContainerStarted","Data":"e06d6d1f41d4df680996c5c7adf8706e59331aeb69d06d0be2bfdd68af2b98cb"}
Nov 01 05:46:03.256776 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.256717    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal" event={"ID":"ef0f634a7979920e9df606f3bb474aac","Type":"ContainerStarted","Data":"9caa216e0fa1aa9115b11851c57c842825c6af055b8bb0e5ec35a0619d90d579"}
Nov 01 05:46:03.256941 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.256901    5619 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:46:03.308861 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.308830    5619 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:46:03.309018 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.308912    5619 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:46:03.320680 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.320656    5619 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 05:46:03.320826 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:03.320706    5619 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:46:03.321465 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.321445    5619 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 05:46:03.321588 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:03.321494    5619 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:46:03.404382 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.404337    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28a06cb4c587db3aac34f3d8afd3bbd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal\" (UID: \"28a06cb4c587db3aac34f3d8afd3bbd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:46:03.404577 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.404396    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28a06cb4c587db3aac34f3d8afd3bbd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal\" (UID: \"28a06cb4c587db3aac34f3d8afd3bbd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:46:03.404577 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.404440    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef0f634a7979920e9df606f3bb474aac-config\") pod \"kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal\" (UID: \"ef0f634a7979920e9df606f3bb474aac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:46:03.504882 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.504846    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28a06cb4c587db3aac34f3d8afd3bbd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal\" (UID: \"28a06cb4c587db3aac34f3d8afd3bbd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:46:03.505071 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.504894    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28a06cb4c587db3aac34f3d8afd3bbd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal\" (UID: \"28a06cb4c587db3aac34f3d8afd3bbd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:46:03.505071 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.504920    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef0f634a7979920e9df606f3bb474aac-config\") pod \"kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal\" (UID: \"ef0f634a7979920e9df606f3bb474aac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:46:03.505071 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.504969    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef0f634a7979920e9df606f3bb474aac-config\") pod \"kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal\" (UID: \"ef0f634a7979920e9df606f3bb474aac\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:46:03.505071 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.504971    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28a06cb4c587db3aac34f3d8afd3bbd7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal\" (UID: \"28a06cb4c587db3aac34f3d8afd3bbd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:46:03.505071 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.504971    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28a06cb4c587db3aac34f3d8afd3bbd7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal\" (UID: \"28a06cb4c587db3aac34f3d8afd3bbd7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:46:03.884742 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.884701    5619 apiserver.go:52] "Watching apiserver"
Nov 01 05:46:03.990490 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.990452    5619 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 05:46:03.991145 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:03.991117    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0","openshift-cluster-node-tuning-operator/tuned-z29gq","openshift-multus/multus-additional-cni-plugins-h4ps2","openshift-image-registry/image-registry-79768fc54c-kxzc9","openshift-image-registry/node-ca-dnkvk","openshift-ingress-canary/ingress-canary-48tvm","openshift-monitoring/monitoring-plugin-77f4f76594-2brlp","openshift-multus/network-metrics-daemon-gjh6r","kube-system/konnectivity-agent-4lm7r","openshift-dns/node-resolver-xqcsn","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vwqd2","openshift-multus/multus-7xz4d","openshift-ovn-kubernetes/ovnkube-node-nt68j","openshift-monitoring/node-exporter-m5vvc","openshift-network-diagnostics/network-check-target-cwn9s","openshift-monitoring/prometheus-k8s-0","openshift-network-operator/iptables-alerter-xv89g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s","openshift-console/downloads-6dd789c8b7-qjp2z","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal","kube-system/global-pull-secret-syncer-jh5gk","kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal","openshift-insights/insights-runtime-extractor-nt6ft","openshift-dns/dns-default-kvs75","openshift-monitoring/metrics-server-69999ffccd-cvcrh"]
Nov 01 05:46:04.017000 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.016973    5619 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 05:46:04.046307 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.046273    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 05:46:04.046307 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.046295    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.046616 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.046591    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.046815 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.046790    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9tjqf\""
Nov 01 05:46:04.047033 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.047009    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 05:46:04.047033 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.047030    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 05:46:04.047198 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.047066    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 05:46:04.047198 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.047140    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.047304 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.047198    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m2d6d\""
Nov 01 05:46:04.047304 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.047211    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.047304 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.047217    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 05:46:04.047304 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.047273    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.047537 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.047472    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-f4d6z\""
Nov 01 05:46:04.047537 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.047486    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.047638 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.047543    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vbc6d\""
Nov 01 05:46:04.047638 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.047552    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.047638 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.047587    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.047638 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.047617    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xlcv8\""
Nov 01 05:46:04.047638 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.047625    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 05:46:04.047638 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.047540    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 05:46:04.047929 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.047711    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 05:46:04.047929 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.047865    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-v8m4f\""
Nov 01 05:46:04.047929 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.047920    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.048080 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.047972    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 05:46:04.048080 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.048048    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 05:46:04.048175 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.048120    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 05:46:04.049128 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.049110    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qfzvw\""
Nov 01 05:46:04.049231 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.049163    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rpsgx\""
Nov 01 05:46:04.049231 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.049190    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-695z8\""
Nov 01 05:46:04.050159 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.050140    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.050373 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.050347    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.050474 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.050384    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.050474 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.050434    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.051674 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.051656    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 05:46:04.051802 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.051785    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.052355 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.052337    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.052454 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.052381    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.052454 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.052396    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 05:46:04.052454 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.052438    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.062491 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.062472    5619 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:46:04.062598 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.062570    5619 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:46:04.096273 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.096174    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.096273 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.096216    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.096273 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.096216    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 01 05:46:04.096521 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.096502    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.096988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.096968    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 01 05:46:04.097090 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.097008    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.097090 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.097042    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.097090 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.097016    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 05:46:04.097090 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.097043    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 05:46:04.097362 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.097337    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 05:46:04.097561 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.097543    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 05:46:04.097638 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.097549    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 05:46:04.100160 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.100127    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 01 05:46:04.103485 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.103462    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 01 05:46:04.103857 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.103832    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 05:46:04.104025 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.103871    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 01 05:46:04.104175 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.103871    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 05:46:04.104175 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.103927    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 01 05:46:04.104259 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.104186    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 01 05:46:04.107399 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.107374    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-registry-tls\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:46:04.107495 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.107410    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-registry-certificates\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:46:04.107495 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.107429    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/95fd5853-f502-4076-90a2-19fb99f52e2c-secret-metrics-server-client-certs\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:46:04.107495 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.107458    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-run-ovn\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.107495 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.107482    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxjd2\" (UniqueName: \"kubernetes.io/projected/04241194-2fed-4174-9eda-e384dcfd4131-kube-api-access-nxjd2\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:46:04.107813 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.107788    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-var-lib-cni-multus\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.108060 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108038    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-registry-certificates\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:46:04.108141 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108056    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d937ef17-8f48-475a-955a-e0599c40d25c-crio-socket\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:46:04.108141 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108097    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:46:04.108229 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108140    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.108229 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108210    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c1d9c47e-944e-4112-9889-b01892353659-env-overrides\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.108412 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108393    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.108412 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108408    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c1d9c47e-944e-4112-9889-b01892353659-env-overrides\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.108503 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108445    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-image-registry-private-configuration\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:46:04.108503 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108483    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.108580 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108509    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/06f0eaf3-04d5-492f-9d3a-6b337091fbea-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-vwqd2\" (UID: \"06f0eaf3-04d5-492f-9d3a-6b337091fbea\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vwqd2"
Nov 01 05:46:04.108580 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108526    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk4fp\" (UniqueName: \"kubernetes.io/projected/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-kube-api-access-wk4fp\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:46:04.108580 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108543    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3df5a1c7-a784-4bec-81d1-5ee5d36c707c-tmp\") pod \"downloads-6dd789c8b7-qjp2z\" (UID: \"3df5a1c7-a784-4bec-81d1-5ee5d36c707c\") " pod="openshift-console/downloads-6dd789c8b7-qjp2z"
Nov 01 05:46:04.108692 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108638    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svf57\" (UniqueName: \"kubernetes.io/projected/3df5a1c7-a784-4bec-81d1-5ee5d36c707c-kube-api-access-svf57\") pod \"downloads-6dd789c8b7-qjp2z\" (UID: \"3df5a1c7-a784-4bec-81d1-5ee5d36c707c\") " pod="openshift-console/downloads-6dd789c8b7-qjp2z"
Nov 01 05:46:04.108692 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108668    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-tmp-dir\") pod \"dns-default-kvs75\" (UID: \"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923\") " pod="openshift-dns/dns-default-kvs75"
Nov 01 05:46:04.108692 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108685    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-var-lib-kubelet\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.108815 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108705    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-run-netns\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.108815 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108727    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-config-volume\") pod \"dns-default-kvs75\" (UID: \"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923\") " pod="openshift-dns/dns-default-kvs75"
Nov 01 05:46:04.108815 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108758    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.108815 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108782    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-web-config\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.108985 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108813    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c1d9c47e-944e-4112-9889-b01892353659-ovnkube-config\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.108985 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108833    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-sysctl-conf\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.108985 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108851    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-installation-pull-secrets\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:46:04.108985 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108880    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-hostroot\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.108985 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108910    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-conf-dir\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.108985 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108936    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2980ba94-fa03-419a-82d7-86405b6178ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:46:04.108985 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108965    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-log-socket\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.109284 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.108989    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-etc-selinux\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:46:04.109284 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.109014    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-cnibin\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.109284 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.109038    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-etc-kubernetes\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.109284 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.109071    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2980ba94-fa03-419a-82d7-86405b6178ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:46:04.109284 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.109093    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.109284 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.109121    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c1d9c47e-944e-4112-9889-b01892353659-ovnkube-config\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.109284 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.109124    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.109284 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.109159    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:46:04.109284 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.109181    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f4eea300-bfd9-465e-9276-16494acaa883-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.109284 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.109198    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.109284 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.109224    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-kubelet-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:46:04.109284 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.109266    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3df5a1c7-a784-4bec-81d1-5ee5d36c707c-tmp\") pod \"downloads-6dd789c8b7-qjp2z\" (UID: \"3df5a1c7-a784-4bec-81d1-5ee5d36c707c\") " pod="openshift-console/downloads-6dd789c8b7-qjp2z"
Nov 01 05:46:04.109932 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.109418    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.109932 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.109480    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-tmp-dir\") pod \"dns-default-kvs75\" (UID: \"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923\") " pod="openshift-dns/dns-default-kvs75"
Nov 01 05:46:04.109932 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.109486    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2980ba94-fa03-419a-82d7-86405b6178ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:46:04.109932 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.109603    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-config-volume\") pod \"dns-default-kvs75\" (UID: \"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923\") " pod="openshift-dns/dns-default-kvs75"
Nov 01 05:46:04.109932 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.109680    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-socket-dir-parent\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.109932 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.109690    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2980ba94-fa03-419a-82d7-86405b6178ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:46:04.109932 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.109737    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-dbus\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:46:04.109932 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.109771    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-original-pull-secret\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:46:04.109932 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.109772    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4eea300-bfd9-465e-9276-16494acaa883-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.109932 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.109908    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.110402 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.109984    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.110402 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110050    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e9a8c0a-32d6-44ef-8a47-f65f6560193d-hosts-file\") pod \"node-resolver-xqcsn\" (UID: \"1e9a8c0a-32d6-44ef-8a47-f65f6560193d\") " pod="openshift-dns/node-resolver-xqcsn"
Nov 01 05:46:04.110402 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110081    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4eea300-bfd9-465e-9276-16494acaa883-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.110402 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110138    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.110402 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110301    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-node-log\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.110402 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110307    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.110402 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110344    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c79lv\" (UniqueName: \"kubernetes.io/projected/c1d9c47e-944e-4112-9889-b01892353659-kube-api-access-c79lv\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.110402 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110375    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-daemon-config\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.110790 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110519    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-daemon-config\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.110790 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110550    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d937ef17-8f48-475a-955a-e0599c40d25c-data-volume\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:46:04.110790 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110606    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d937ef17-8f48-475a-955a-e0599c40d25c-data-volume\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:46:04.110790 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110638    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfpq7\" (UniqueName: \"kubernetes.io/projected/e5ace06a-0f90-46ef-8a04-fdc0f7180a1e-kube-api-access-jfpq7\") pod \"node-ca-dnkvk\" (UID: \"e5ace06a-0f90-46ef-8a04-fdc0f7180a1e\") " pod="openshift-image-registry/node-ca-dnkvk"
Nov 01 05:46:04.110790 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110689    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-config-volume\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.110790 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110717    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.110790 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110748    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/05e09ea1-324b-49ef-a4a5-2645b50232a4-agent-certs\") pod \"konnectivity-agent-4lm7r\" (UID: \"05e09ea1-324b-49ef-a4a5-2645b50232a4\") " pod="kube-system/konnectivity-agent-4lm7r"
Nov 01 05:46:04.110790 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110769    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-sys\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.111181 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110795    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7ttp\" (UniqueName: \"kubernetes.io/projected/dc7de5f7-34ee-46ae-81eb-81717d8f1a52-kube-api-access-w7ttp\") pod \"ingress-canary-48tvm\" (UID: \"dc7de5f7-34ee-46ae-81eb-81717d8f1a52\") " pod="openshift-ingress-canary/ingress-canary-48tvm"
Nov 01 05:46:04.111181 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110819    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-ca-trust-extracted\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:46:04.111181 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110839    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs\") pod \"network-metrics-daemon-gjh6r\" (UID: \"a18b7694-133e-4663-b8f6-a36f981ab717\") " pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:46:04.111181 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110865    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/941ae7ae-e26d-4adb-8d74-729e4c038861-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2brlp\" (UID: \"941ae7ae-e26d-4adb-8d74-729e4c038861\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2brlp"
Nov 01 05:46:04.111181 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110889    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-kubelet-config\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:46:04.111181 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110916    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.111181 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110940    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwg57\" (UniqueName: \"kubernetes.io/projected/a18b7694-133e-4663-b8f6-a36f981ab717-kube-api-access-kwg57\") pod \"network-metrics-daemon-gjh6r\" (UID: \"a18b7694-133e-4663-b8f6-a36f981ab717\") " pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:46:04.111181 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.110980    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-sys\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:46:04.111181 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.111015    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-trusted-ca\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:46:04.111181 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.111040    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4eea300-bfd9-465e-9276-16494acaa883-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.111181 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.111078    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkn77\" (UniqueName: \"kubernetes.io/projected/f4eea300-bfd9-465e-9276-16494acaa883-kube-api-access-bkn77\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.111181 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.111102    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-config\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.111181 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.111109    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-ca-trust-extracted\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:46:04.111181 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.111127    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-run\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.111181 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.111167    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhq5k\" (UniqueName: \"kubernetes.io/projected/95fd5853-f502-4076-90a2-19fb99f52e2c-kube-api-access-vhq5k\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:46:04.111923 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.111207    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2980ba94-fa03-419a-82d7-86405b6178ac-system-cni-dir\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:46:04.111923 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.111241    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2980ba94-fa03-419a-82d7-86405b6178ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:46:04.111923 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.111276    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5ace06a-0f90-46ef-8a04-fdc0f7180a1e-host\") pod \"node-ca-dnkvk\" (UID: \"e5ace06a-0f90-46ef-8a04-fdc0f7180a1e\") " pod="openshift-image-registry/node-ca-dnkvk"
Nov 01 05:46:04.111923 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.111397    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-tls\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:46:04.111923 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.111430    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2980ba94-fa03-419a-82d7-86405b6178ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:46:04.111923 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.111709    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/05e09ea1-324b-49ef-a4a5-2645b50232a4-agent-certs\") pod \"konnectivity-agent-4lm7r\" (UID: \"05e09ea1-324b-49ef-a4a5-2645b50232a4\") " pod="kube-system/konnectivity-agent-4lm7r"
Nov 01 05:46:04.112473 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.111460    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsndz\" (UniqueName: \"kubernetes.io/projected/1e9a8c0a-32d6-44ef-8a47-f65f6560193d-kube-api-access-tsndz\") pod \"node-resolver-xqcsn\" (UID: \"1e9a8c0a-32d6-44ef-8a47-f65f6560193d\") " pod="openshift-dns/node-resolver-xqcsn"
Nov 01 05:46:04.112604 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.112573    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-kubelet\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.112604 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.112581    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4eea300-bfd9-465e-9276-16494acaa883-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.112768 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.112745    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-trusted-ca\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:46:04.112846 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.112771    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2980ba94-fa03-419a-82d7-86405b6178ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:46:04.112892 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.112871    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.112945 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.112902    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-var-lib-kubelet\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.112988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.112928    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f4eea300-bfd9-465e-9276-16494acaa883-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.112988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.112973    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-systemd-units\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.113059 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.112991    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95fd5853-f502-4076-90a2-19fb99f52e2c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:46:04.113059 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113007    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-accelerators-collector-config\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:46:04.113134 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113065    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-web-config\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.113134 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113095    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rprhl\" (UniqueName: \"kubernetes.io/projected/9efa148c-f7c3-4f9d-bee3-cc1eb353b808-kube-api-access-rprhl\") pod \"iptables-alerter-xv89g\" (UID: \"9efa148c-f7c3-4f9d-bee3-cc1eb353b808\") " pod="openshift-network-operator/iptables-alerter-xv89g"
Nov 01 05:46:04.113134 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113111    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f4eea300-bfd9-465e-9276-16494acaa883-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.113134 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113130    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.113351 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113252    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-lib-modules\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.113351 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113279    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-sys-fs\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:46:04.113351 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113310    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95fd5853-f502-4076-90a2-19fb99f52e2c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:46:04.113351 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113315    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.113557 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113395    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-accelerators-collector-config\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:46:04.113557 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113397    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-run-ovn-kubernetes\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.113557 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113440    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwxk7\" (UniqueName: \"kubernetes.io/projected/bdeea076-cd20-459e-b1a5-79ec825e95e3-kube-api-access-nwxk7\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.113557 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113467    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.113557 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113484    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-systemd\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.113557 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113503    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0e384e0-dcf6-410c-b344-8d797c57e1c9-tmp\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.113557 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113544    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d937ef17-8f48-475a-955a-e0599c40d25c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:46:04.113834 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113669    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.113834 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113698    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stntj\" (UniqueName: \"kubernetes.io/projected/941ae7ae-e26d-4adb-8d74-729e4c038861-kube-api-access-stntj\") pod \"monitoring-plugin-77f4f76594-2brlp\" (UID: \"941ae7ae-e26d-4adb-8d74-729e4c038861\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2brlp"
Nov 01 05:46:04.113834 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113715    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0e384e0-dcf6-410c-b344-8d797c57e1c9-tmp\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.113834 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113719    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9efa148c-f7c3-4f9d-bee3-cc1eb353b808-iptables-alerter-script\") pod \"iptables-alerter-xv89g\" (UID: \"9efa148c-f7c3-4f9d-bee3-cc1eb353b808\") " pod="openshift-network-operator/iptables-alerter-xv89g"
Nov 01 05:46:04.114003 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113872    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.114003 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113911    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp4hf\" (UniqueName: \"kubernetes.io/projected/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-kube-api-access-rp4hf\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.114003 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113878    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9efa148c-f7c3-4f9d-bee3-cc1eb353b808-iptables-alerter-script\") pod \"iptables-alerter-xv89g\" (UID: \"9efa148c-f7c3-4f9d-bee3-cc1eb353b808\") " pod="openshift-network-operator/iptables-alerter-xv89g"
Nov 01 05:46:04.114003 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113941    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-sysctl-d\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.114003 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.113982    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-registration-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:46:04.114216 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114009    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58hcx\" (UniqueName: \"kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx\") pod \"network-check-target-cwn9s\" (UID: \"a78d89dc-2ed6-4a2e-9558-f54b73a806d2\") " pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:46:04.114216 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114025    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-root\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:46:04.114216 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114041    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.114216 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114067    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-run-netns\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.114216 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114089    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-modprobe-d\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.114216 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114113    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86v89\" (UniqueName: \"kubernetes.io/projected/d0e384e0-dcf6-410c-b344-8d797c57e1c9-kube-api-access-86v89\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.114216 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114139    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc7de5f7-34ee-46ae-81eb-81717d8f1a52-cert\") pod \"ingress-canary-48tvm\" (UID: \"dc7de5f7-34ee-46ae-81eb-81717d8f1a52\") " pod="openshift-ingress-canary/ingress-canary-48tvm"
Nov 01 05:46:04.114216 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114162    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2980ba94-fa03-419a-82d7-86405b6178ac-cnibin\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:46:04.114216 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114184    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9efa148c-f7c3-4f9d-bee3-cc1eb353b808-host-slash\") pod \"iptables-alerter-xv89g\" (UID: \"9efa148c-f7c3-4f9d-bee3-cc1eb353b808\") " pod="openshift-network-operator/iptables-alerter-xv89g"
Nov 01 05:46:04.114216 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114200    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-run-systemd\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.114707 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114236    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdeea076-cd20-459e-b1a5-79ec825e95e3-cni-binary-copy\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.114707 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114261    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-run-k8s-cni-cncf-io\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.114707 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114291    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/95fd5853-f502-4076-90a2-19fb99f52e2c-metrics-server-audit-profiles\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:46:04.114707 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114314    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg2zd\" (UniqueName: \"kubernetes.io/projected/2980ba94-fa03-419a-82d7-86405b6178ac-kube-api-access-vg2zd\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:46:04.114707 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114353    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-textfile\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:46:04.114707 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114397    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-metrics-client-ca\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:46:04.114707 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114462    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-textfile\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:46:04.114707 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114466    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.114707 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114666    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f4eea300-bfd9-465e-9276-16494acaa883-config-out\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.114707 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114702    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-bound-sa-token\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:46:04.115142 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114731    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-device-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:46:04.115142 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114741    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdeea076-cd20-459e-b1a5-79ec825e95e3-cni-binary-copy\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.115142 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114751    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-metrics-client-ca\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:46:04.115142 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114756    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-var-lib-cni-bin\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.115142 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114854    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/95fd5853-f502-4076-90a2-19fb99f52e2c-metrics-server-audit-profiles\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:46:04.115142 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114918    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-wtmp\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:46:04.115142 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114946    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f4eea300-bfd9-465e-9276-16494acaa883-config-out\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.115142 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.114961    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.115142 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.115010    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.115142 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.115107    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-var-lib-openvswitch\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.115142 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.115136    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-cni-netd\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.115630 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.115161    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-run-multus-certs\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.115630 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.115192    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hhd5\" (UniqueName: \"kubernetes.io/projected/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-kube-api-access-2hhd5\") pod \"dns-default-kvs75\" (UID: \"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923\") " pod="openshift-dns/dns-default-kvs75"
Nov 01 05:46:04.115630 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.115229    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-config-out\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.115630 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.115298    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-config-out\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.115630 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.115308    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.115630 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.115364    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.115630 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.115414    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.115630 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.115556    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.115630 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.115617    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-slash\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.115976 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.115648    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-cni-bin\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.115976 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.115668    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-tuned\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.115976 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.115692    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-system-cni-dir\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.115976 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.115742    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-metrics-tls\") pod \"dns-default-kvs75\" (UID: \"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923\") " pod="openshift-dns/dns-default-kvs75"
Nov 01 05:46:04.115976 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.115788    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-tuned\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.115976 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.115850    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-socket-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:46:04.115976 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.115918    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/95fd5853-f502-4076-90a2-19fb99f52e2c-secret-metrics-server-tls\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:46:04.115976 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.115950    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/95fd5853-f502-4076-90a2-19fb99f52e2c-client-ca-bundle\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:46:04.116284 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.116021    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5ace06a-0f90-46ef-8a04-fdc0f7180a1e-serviceca\") pod \"node-ca-dnkvk\" (UID: \"e5ace06a-0f90-46ef-8a04-fdc0f7180a1e\") " pod="openshift-image-registry/node-ca-dnkvk"
Nov 01 05:46:04.116284 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.116070    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e9a8c0a-32d6-44ef-8a47-f65f6560193d-tmp-dir\") pod \"node-resolver-xqcsn\" (UID: \"1e9a8c0a-32d6-44ef-8a47-f65f6560193d\") " pod="openshift-dns/node-resolver-xqcsn"
Nov 01 05:46:04.116284 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.116144    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e9a8c0a-32d6-44ef-8a47-f65f6560193d-tmp-dir\") pod \"node-resolver-xqcsn\" (UID: \"1e9a8c0a-32d6-44ef-8a47-f65f6560193d\") " pod="openshift-dns/node-resolver-xqcsn"
Nov 01 05:46:04.116284 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.116173    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-sysconfig\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.116284 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.116277    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-kubernetes\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.116540 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.116318    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d937ef17-8f48-475a-955a-e0599c40d25c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:46:04.116540 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.116384    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-etc-openvswitch\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.116540 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.116422    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/05e09ea1-324b-49ef-a4a5-2645b50232a4-konnectivity-ca\") pod \"konnectivity-agent-4lm7r\" (UID: \"05e09ea1-324b-49ef-a4a5-2645b50232a4\") " pod="kube-system/konnectivity-agent-4lm7r"
Nov 01 05:46:04.116703 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.116584    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-cni-dir\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.116703 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.116590    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d937ef17-8f48-475a-955a-e0599c40d25c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:46:04.116703 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.116650    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlqr8\" (UniqueName: \"kubernetes.io/projected/d937ef17-8f48-475a-955a-e0599c40d25c-kube-api-access-zlqr8\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:46:04.116703 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.116680    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/05e09ea1-324b-49ef-a4a5-2645b50232a4-konnectivity-ca\") pod \"konnectivity-agent-4lm7r\" (UID: \"05e09ea1-324b-49ef-a4a5-2645b50232a4\") " pod="kube-system/konnectivity-agent-4lm7r"
Nov 01 05:46:04.116896 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.116725    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlj5f\" (UniqueName: \"kubernetes.io/projected/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-kube-api-access-zlj5f\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:46:04.116896 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.116744    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5ace06a-0f90-46ef-8a04-fdc0f7180a1e-serviceca\") pod \"node-ca-dnkvk\" (UID: \"e5ace06a-0f90-46ef-8a04-fdc0f7180a1e\") " pod="openshift-image-registry/node-ca-dnkvk"
Nov 01 05:46:04.116896 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.116756    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.116896 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.116788    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/95fd5853-f502-4076-90a2-19fb99f52e2c-audit-log\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:46:04.116896 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.116808    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-run-openvswitch\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.116896 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.116892    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1d9c47e-944e-4112-9889-b01892353659-ovn-node-metrics-cert\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.117182 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.116913    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c1d9c47e-944e-4112-9889-b01892353659-ovnkube-script-lib\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.117182 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.116944    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-host\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.117182 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.117030    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/95fd5853-f502-4076-90a2-19fb99f52e2c-audit-log\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:46:04.117299 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.117185    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1d9c47e-944e-4112-9889-b01892353659-ovn-node-metrics-cert\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.117299 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.117217    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c1d9c47e-944e-4112-9889-b01892353659-ovnkube-script-lib\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.147943 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.147877    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 01 05:46:04.148100 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.148083    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 01 05:46:04.148393 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.148373    5619 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 05:46:04.148524 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.148395    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 05:46:04.148524 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.148453    5619 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 05:46:04.148524 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:04.148508    5619 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:46:04.148722 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.148682    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 01 05:46:04.148883 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:04.148861    5619 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal"
Nov 01 05:46:04.148974 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.148907    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 05:46:04.149338 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149309    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 01 05:46:04.149413 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149312    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-87mzr\""
Nov 01 05:46:04.149542 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149525    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 01 05:46:04.149590 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149574    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 01 05:46:04.149625 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149597    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 01 05:46:04.149675 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149654    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vp9l2\""
Nov 01 05:46:04.149675 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149664    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vdjzw\""
Nov 01 05:46:04.149756 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149681    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 01 05:46:04.149756 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149714    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 01 05:46:04.149756 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149737    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 01 05:46:04.149854 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149758    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 01 05:46:04.149854 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149783    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 01 05:46:04.149854 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149807    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-75cn8\""
Nov 01 05:46:04.149854 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149828    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dmpoc73dr8cgo\""
Nov 01 05:46:04.149854 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149736    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f427j\""
Nov 01 05:46:04.149854 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149844    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 01 05:46:04.149854 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149826    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rz4ht\""
Nov 01 05:46:04.150123 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149879    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 01 05:46:04.150123 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149916    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jzphk\""
Nov 01 05:46:04.150123 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149926    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9hwbb\""
Nov 01 05:46:04.150123 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149951    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 05:46:04.150123 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149956    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 01 05:46:04.150123 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149971    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-z8gwd\""
Nov 01 05:46:04.150123 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149988    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 01 05:46:04.150123 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149996    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 01 05:46:04.150123 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.150009    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 01 05:46:04.150123 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.150048    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dj5fv\""
Nov 01 05:46:04.150123 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.150053    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 01 05:46:04.150123 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.149990    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 01 05:46:04.150123 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.150119    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 05:46:04.150591 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.150133    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 05:46:04.150591 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.150139    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-t2x2d\""
Nov 01 05:46:04.150591 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.150172    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 05:46:04.150591 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.150187    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 05:46:04.150591 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.150205    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 01 05:46:04.150591 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.150236    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-74ptq\""
Nov 01 05:46:04.150591 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.150239    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p4xg8\""
Nov 01 05:46:04.150591 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.150297    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 05:46:04.150591 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.150306    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 05:46:04.150591 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.150299    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5zklw\""
Nov 01 05:46:04.150591 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.150361    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 01 05:46:04.150591 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.150437    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-12tgr4asi4bd8\""
Nov 01 05:46:04.150591 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.150506    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 05:46:04.151154 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.150653    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f4eea300-bfd9-465e-9276-16494acaa883-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.151154 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.150812    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-installation-pull-secrets\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:46:04.151154 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.150959    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.151154 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.151144    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-config-volume\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.151304 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.151288    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.151378 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.151298    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.151461 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.151440    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-image-registry-private-configuration\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:46:04.151558 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.151470    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.151620 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.151581    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a18b7694-133e-4663-b8f6-a36f981ab717-metrics-certs\") pod \"network-metrics-daemon-gjh6r\" (UID: \"a18b7694-133e-4663-b8f6-a36f981ab717\") " pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:46:04.151676 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.151631    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.151766 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.151749    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:46:04.151808 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.151755    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/941ae7ae-e26d-4adb-8d74-729e4c038861-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2brlp\" (UID: \"941ae7ae-e26d-4adb-8d74-729e4c038861\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2brlp"
Nov 01 05:46:04.151907 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.151883    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/06f0eaf3-04d5-492f-9d3a-6b337091fbea-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-vwqd2\" (UID: \"06f0eaf3-04d5-492f-9d3a-6b337091fbea\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vwqd2"
Nov 01 05:46:04.151959 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.151922    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-registry-tls\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:46:04.152055 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.152039    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-web-config\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.153498 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.153204    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-config\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.153498 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.153217    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.153498 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.153352    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-web-config\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.153498 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.153406    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-tls\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:46:04.153498 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.153472    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.153680 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.153585    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.154476 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.154456    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.154553 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.154505    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d937ef17-8f48-475a-955a-e0599c40d25c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:46:04.154656 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.154633    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc7de5f7-34ee-46ae-81eb-81717d8f1a52-cert\") pod \"ingress-canary-48tvm\" (UID: \"dc7de5f7-34ee-46ae-81eb-81717d8f1a52\") " pod="openshift-ingress-canary/ingress-canary-48tvm"
Nov 01 05:46:04.155622 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.155600    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.155723 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.155664    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.155723 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.155708    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.155795 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.155753    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.156911 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.156891    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/95fd5853-f502-4076-90a2-19fb99f52e2c-client-ca-bundle\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:46:04.157011 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.156984    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/95fd5853-f502-4076-90a2-19fb99f52e2c-secret-metrics-server-tls\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:46:04.157011 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.156984    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-metrics-tls\") pod \"dns-default-kvs75\" (UID: \"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923\") " pod="openshift-dns/dns-default-kvs75"
Nov 01 05:46:04.157117 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.157098    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.158090 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.158070    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/95fd5853-f502-4076-90a2-19fb99f52e2c-secret-metrics-server-client-certs\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:46:04.168858 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.168833    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxjd2\" (UniqueName: \"kubernetes.io/projected/04241194-2fed-4174-9eda-e384dcfd4131-kube-api-access-nxjd2\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:46:04.170833 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.170797    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwg57\" (UniqueName: \"kubernetes.io/projected/a18b7694-133e-4663-b8f6-a36f981ab717-kube-api-access-kwg57\") pod \"network-metrics-daemon-gjh6r\" (UID: \"a18b7694-133e-4663-b8f6-a36f981ab717\") " pod="openshift-multus/network-metrics-daemon-gjh6r"
Nov 01 05:46:04.170947 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.170866    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhq5k\" (UniqueName: \"kubernetes.io/projected/95fd5853-f502-4076-90a2-19fb99f52e2c-kube-api-access-vhq5k\") pod \"metrics-server-69999ffccd-cvcrh\" (UID: \"95fd5853-f502-4076-90a2-19fb99f52e2c\") " pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:46:04.170947 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.170872    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk4fp\" (UniqueName: \"kubernetes.io/projected/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-kube-api-access-wk4fp\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:46:04.171295 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.171271    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c79lv\" (UniqueName: \"kubernetes.io/projected/c1d9c47e-944e-4112-9889-b01892353659-kube-api-access-c79lv\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.171497 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.171476    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfpq7\" (UniqueName: \"kubernetes.io/projected/e5ace06a-0f90-46ef-8a04-fdc0f7180a1e-kube-api-access-jfpq7\") pod \"node-ca-dnkvk\" (UID: \"e5ace06a-0f90-46ef-8a04-fdc0f7180a1e\") " pod="openshift-image-registry/node-ca-dnkvk"
Nov 01 05:46:04.171634 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.171614    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svf57\" (UniqueName: \"kubernetes.io/projected/3df5a1c7-a784-4bec-81d1-5ee5d36c707c-kube-api-access-svf57\") pod \"downloads-6dd789c8b7-qjp2z\" (UID: \"3df5a1c7-a784-4bec-81d1-5ee5d36c707c\") " pod="openshift-console/downloads-6dd789c8b7-qjp2z"
Nov 01 05:46:04.171981 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.171958    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7ttp\" (UniqueName: \"kubernetes.io/projected/dc7de5f7-34ee-46ae-81eb-81717d8f1a52-kube-api-access-w7ttp\") pod \"ingress-canary-48tvm\" (UID: \"dc7de5f7-34ee-46ae-81eb-81717d8f1a52\") " pod="openshift-ingress-canary/ingress-canary-48tvm"
Nov 01 05:46:04.172122 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.172106    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsndz\" (UniqueName: \"kubernetes.io/projected/1e9a8c0a-32d6-44ef-8a47-f65f6560193d-kube-api-access-tsndz\") pod \"node-resolver-xqcsn\" (UID: \"1e9a8c0a-32d6-44ef-8a47-f65f6560193d\") " pod="openshift-dns/node-resolver-xqcsn"
Nov 01 05:46:04.172765 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.172746    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rprhl\" (UniqueName: \"kubernetes.io/projected/9efa148c-f7c3-4f9d-bee3-cc1eb353b808-kube-api-access-rprhl\") pod \"iptables-alerter-xv89g\" (UID: \"9efa148c-f7c3-4f9d-bee3-cc1eb353b808\") " pod="openshift-network-operator/iptables-alerter-xv89g"
Nov 01 05:46:04.173626 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.173596    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkn77\" (UniqueName: \"kubernetes.io/projected/f4eea300-bfd9-465e-9276-16494acaa883-kube-api-access-bkn77\") pod \"alertmanager-main-0\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:04.173702 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.173659    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwxk7\" (UniqueName: \"kubernetes.io/projected/bdeea076-cd20-459e-b1a5-79ec825e95e3-kube-api-access-nwxk7\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.174581 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.174560    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stntj\" (UniqueName: \"kubernetes.io/projected/941ae7ae-e26d-4adb-8d74-729e4c038861-kube-api-access-stntj\") pod \"monitoring-plugin-77f4f76594-2brlp\" (UID: \"941ae7ae-e26d-4adb-8d74-729e4c038861\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2brlp"
Nov 01 05:46:04.175779 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.175761    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp4hf\" (UniqueName: \"kubernetes.io/projected/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-kube-api-access-rp4hf\") pod \"prometheus-k8s-0\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:04.176662 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.176638    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg2zd\" (UniqueName: \"kubernetes.io/projected/2980ba94-fa03-419a-82d7-86405b6178ac-kube-api-access-vg2zd\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:46:04.176938 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.176917    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86v89\" (UniqueName: \"kubernetes.io/projected/d0e384e0-dcf6-410c-b344-8d797c57e1c9-kube-api-access-86v89\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.177163 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.177139    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58hcx\" (UniqueName: \"kubernetes.io/projected/a78d89dc-2ed6-4a2e-9558-f54b73a806d2-kube-api-access-58hcx\") pod \"network-check-target-cwn9s\" (UID: \"a78d89dc-2ed6-4a2e-9558-f54b73a806d2\") " pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:46:04.177500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.177476    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa8083b6-e023-46fe-bdb3-a9e99079eb3d-bound-sa-token\") pod \"image-registry-79768fc54c-kxzc9\" (UID: \"aa8083b6-e023-46fe-bdb3-a9e99079eb3d\") " pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:46:04.177573 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.177537    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlqr8\" (UniqueName: \"kubernetes.io/projected/d937ef17-8f48-475a-955a-e0599c40d25c-kube-api-access-zlqr8\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:46:04.177680 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.177658    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hhd5\" (UniqueName: \"kubernetes.io/projected/910a0fc6-8aa8-4b22-b767-b0ecf3b6c923-kube-api-access-2hhd5\") pod \"dns-default-kvs75\" (UID: \"910a0fc6-8aa8-4b22-b767-b0ecf3b6c923\") " pod="openshift-dns/dns-default-kvs75"
Nov 01 05:46:04.178151 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.178130    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlj5f\" (UniqueName: \"kubernetes.io/projected/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-kube-api-access-zlj5f\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:46:04.217391 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.217355    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-kubelet\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.217391 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.217401    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.217391 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.217426    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-var-lib-kubelet\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.217391 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.217521    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.217391 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.217568    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-systemd-units\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.217391 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.217583    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-var-lib-kubelet\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.217998 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.217618    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-kubelet\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.217998 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.217671    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-lib-modules\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.217998 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.217705    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-sys-fs\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:46:04.217998 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.217732    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-run-ovn-kubernetes\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.217998 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.217742    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-systemd-units\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.217998 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.217795    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-systemd\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.217998 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.217748    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-systemd\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.217998 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.217806    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-run-ovn-kubernetes\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.217998 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.217814    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-sys-fs\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:46:04.217998 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.217797    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-lib-modules\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.217998 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.217887    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-sysctl-d\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.217998 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.217928    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-registration-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:46:04.217998 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.217952    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-root\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:46:04.217998 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.217971    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-run-netns\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.217998 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.217983    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-sysctl-d\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.217998 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.217997    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-modprobe-d\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218009    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-root\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218013    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-registration-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218023    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2980ba94-fa03-419a-82d7-86405b6178ac-cnibin\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218063    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2980ba94-fa03-419a-82d7-86405b6178ac-cnibin\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218066    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9efa148c-f7c3-4f9d-bee3-cc1eb353b808-host-slash\") pod \"iptables-alerter-xv89g\" (UID: \"9efa148c-f7c3-4f9d-bee3-cc1eb353b808\") " pod="openshift-network-operator/iptables-alerter-xv89g"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218031    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-run-netns\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218097    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-run-systemd\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218114    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-modprobe-d\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218117    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-run-k8s-cni-cncf-io\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218149    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-run-k8s-cni-cncf-io\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218155    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-device-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218159    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9efa148c-f7c3-4f9d-bee3-cc1eb353b808-host-slash\") pod \"iptables-alerter-xv89g\" (UID: \"9efa148c-f7c3-4f9d-bee3-cc1eb353b808\") " pod="openshift-network-operator/iptables-alerter-xv89g"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218164    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-run-systemd\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218181    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-var-lib-cni-bin\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218211    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-wtmp\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218217    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-device-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218238    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-var-lib-openvswitch\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218246    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-var-lib-cni-bin\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218264    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-cni-netd\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218273    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-node-exporter-wtmp\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218285    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-var-lib-openvswitch\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218288    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-run-multus-certs\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218313    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-cni-netd\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218333    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-run-multus-certs\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218341    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-slash\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218366    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-cni-bin\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218391    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-system-cni-dir\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218400    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-slash\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218413    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-socket-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218444    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-sysconfig\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218457    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-system-cni-dir\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218468    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-kubernetes\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218499    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-socket-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218500    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-sysconfig\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218502    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-etc-openvswitch\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218537    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-kubernetes\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218544    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-etc-openvswitch\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218561    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-cni-dir\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218444    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-host-cni-bin\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218600    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-run-openvswitch\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218623    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-host\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218647    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-run-ovn\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218654    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-run-openvswitch\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218671    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-var-lib-cni-multus\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218698    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-var-lib-cni-multus\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218666    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-cni-dir\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218706    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d937ef17-8f48-475a-955a-e0599c40d25c-crio-socket\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218704    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-run-ovn\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.218727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218741    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-host\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218771    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d937ef17-8f48-475a-955a-e0599c40d25c-crio-socket\") pod \"insights-runtime-extractor-nt6ft\" (UID: \"d937ef17-8f48-475a-955a-e0599c40d25c\") " pod="openshift-insights/insights-runtime-extractor-nt6ft"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218778    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-var-lib-kubelet\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218797    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-run-netns\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218815    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-sysctl-conf\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218828    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-var-lib-kubelet\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218840    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-hostroot\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218863    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-conf-dir\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218889    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-log-socket\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218914    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-etc-selinux\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218920    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-conf-dir\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218922    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-hostroot\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218868    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-host-run-netns\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218966    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-log-socket\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218937    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-cnibin\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218981    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-cnibin\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.218990    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-etc-sysctl-conf\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219001    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-etc-kubernetes\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219008    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-etc-selinux\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219031    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-etc-kubernetes\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219034    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-kubelet-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219064    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-socket-dir-parent\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219087    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-dbus\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219109    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04241194-2fed-4174-9eda-e384dcfd4131-kubelet-dir\") pod \"aws-ebs-csi-driver-node-27c6s\" (UID: \"04241194-2fed-4174-9eda-e384dcfd4131\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-27c6s"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219113    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e9a8c0a-32d6-44ef-8a47-f65f6560193d-hosts-file\") pod \"node-resolver-xqcsn\" (UID: \"1e9a8c0a-32d6-44ef-8a47-f65f6560193d\") " pod="openshift-dns/node-resolver-xqcsn"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219140    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-node-log\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219152    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bdeea076-cd20-459e-b1a5-79ec825e95e3-multus-socket-dir-parent\") pod \"multus-7xz4d\" (UID: \"bdeea076-cd20-459e-b1a5-79ec825e95e3\") " pod="openshift-multus/multus-7xz4d"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219157    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e9a8c0a-32d6-44ef-8a47-f65f6560193d-hosts-file\") pod \"node-resolver-xqcsn\" (UID: \"1e9a8c0a-32d6-44ef-8a47-f65f6560193d\") " pod="openshift-dns/node-resolver-xqcsn"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219165    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-sys\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219195    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1d9c47e-944e-4112-9889-b01892353659-node-log\") pod \"ovnkube-node-nt68j\" (UID: \"c1d9c47e-944e-4112-9889-b01892353659\") " pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219193    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-kubelet-config\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219210    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-dbus\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219231    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-sys\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219245    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-sys\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219254    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-run\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219261    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7f767e63-7cc7-4b5e-9941-860fe5f4cd62-kubelet-config\") pod \"global-pull-secret-syncer-jh5gk\" (UID: \"7f767e63-7cc7-4b5e-9941-860fe5f4cd62\") " pod="kube-system/global-pull-secret-syncer-jh5gk"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219277    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2980ba94-fa03-419a-82d7-86405b6178ac-system-cni-dir\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219286    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec642c5b-c364-4faa-94d5-bb5bf32a2cb8-sys\") pod \"node-exporter-m5vvc\" (UID: \"ec642c5b-c364-4faa-94d5-bb5bf32a2cb8\") " pod="openshift-monitoring/node-exporter-m5vvc"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219314    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2980ba94-fa03-419a-82d7-86405b6178ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219360    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2980ba94-fa03-419a-82d7-86405b6178ac-system-cni-dir\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219361    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5ace06a-0f90-46ef-8a04-fdc0f7180a1e-host\") pod \"node-ca-dnkvk\" (UID: \"e5ace06a-0f90-46ef-8a04-fdc0f7180a1e\") " pod="openshift-image-registry/node-ca-dnkvk"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219401    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2980ba94-fa03-419a-82d7-86405b6178ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h4ps2\" (UID: \"2980ba94-fa03-419a-82d7-86405b6178ac\") " pod="openshift-multus/multus-additional-cni-plugins-h4ps2"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219340    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0e384e0-dcf6-410c-b344-8d797c57e1c9-run\") pod \"tuned-z29gq\" (UID: \"d0e384e0-dcf6-410c-b344-8d797c57e1c9\") " pod="openshift-cluster-node-tuning-operator/tuned-z29gq"
Nov 01 05:46:04.220758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.219415    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5ace06a-0f90-46ef-8a04-fdc0f7180a1e-host\") pod \"node-ca-dnkvk\" (UID: \"e5ace06a-0f90-46ef-8a04-fdc0f7180a1e\") " pod="openshift-image-registry/node-ca-dnkvk"
Nov 01 05:46:04.309349 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.309313    5619 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 05:46:04.348478 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.348350    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:04.733299 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:04.733269    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kvs75"
Nov 01 05:46:05.330883 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:05.330838    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:05.477759 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:05.477681    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:05.816212 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:05.816113    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-qjp2z"
Nov 01 05:46:05.820247 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:05.820218    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-qjp2z"
Nov 01 05:46:06.875172 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:06.875139    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:06.875618 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:06.875456    5619 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 05:46:08.363131 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:08.363083    5619 patch_prober.go:29] interesting pod/metrics-server-69999ffccd-cvcrh container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.133.0.14:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 05:46:08.363644 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:08.363182    5619 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh" podUID="95fd5853-f502-4076-90a2-19fb99f52e2c" containerName="metrics-server" probeResult="failure" output="Get \"https://10.133.0.14:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 05:46:09.091758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:09.091728    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4254b02e-3f9a-4bcc-b711-de1bbb5b3395","Type":"ContainerStarted","Data":"2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79"}
Nov 01 05:46:09.707245 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:09.707206    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:09.795518 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:09.795485    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2brlp"
Nov 01 05:46:09.799656 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:09.799632    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2brlp"
Nov 01 05:46:09.859755 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:09.859710    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:10.099010 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:10.098969    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4254b02e-3f9a-4bcc-b711-de1bbb5b3395","Type":"ContainerStarted","Data":"81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40"}
Nov 01 05:46:11.069829 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:11.069681    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:11.236041 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:11.236006    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:11.380086 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:11.380052    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:11.876411 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:11.876380    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79768fc54c-kxzc9"
Nov 01 05:46:11.949720 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:11.949693    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:46:12.158061 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:12.157963    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:12.307365 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:12.307307    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:12.476890 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:12.476859    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:12.603505 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:12.603482    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:46:12.619597 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:12.619561    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:12.960572 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:12.960503    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vwqd2"
Nov 01 05:46:13.116387 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:13.116354    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4254b02e-3f9a-4bcc-b711-de1bbb5b3395","Type":"ContainerStarted","Data":"366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7"}
Nov 01 05:46:13.116559 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:13.116417    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4254b02e-3f9a-4bcc-b711-de1bbb5b3395","Type":"ContainerStarted","Data":"37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e"}
Nov 01 05:46:13.116559 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:13.116442    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4254b02e-3f9a-4bcc-b711-de1bbb5b3395","Type":"ContainerStarted","Data":"f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894"}
Nov 01 05:46:13.116559 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:13.116465    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4254b02e-3f9a-4bcc-b711-de1bbb5b3395","Type":"ContainerStarted","Data":"db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a"}
Nov 01 05:46:14.320104 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:14.320074    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:14.462917 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:14.462879    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:16.054159 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:16.054126    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:16.443904 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:16.443871    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:16.632561 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:16.632523    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:16.775558 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:16.775482    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:18.868482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:18.868452    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nt68j"
Nov 01 05:46:21.582420 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:21.582387    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:21.769831 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:21.769798    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:23.338209 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:23.338179    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:46:24.703998 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:24.703965    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:24.843426 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:24.843394    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:25.878650 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:25.878620    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:46:27.827959 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:27.827925    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4lm7r"
Nov 01 05:46:31.096260 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:31.096220    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:31.286679 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:31.286639    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:31.431627 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:31.431556    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:31.952431 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:31.952397    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cwn9s"
Nov 01 05:46:32.760618 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:32.760581    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:32.814002 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:32.813979    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:33.205966 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:33.205941    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:39.027278 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:39.027224    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:39.168554 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:39.168515    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:42.936740 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:42.936691    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:43.162164 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:43.162117    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:43.343591 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:43.343568    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:46:43.347401 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:43.347379    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-69999ffccd-cvcrh"
Nov 01 05:46:45.132753 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:45.132724    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 05:46:45.133171 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:45.133143    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="alertmanager" containerID="cri-o://6de0ae75117fddfa787c7ff5deb8d4ad1383a2937cdf7c5fc502728190ea31a1" gracePeriod=120
Nov 01 05:46:45.133238 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:45.133173    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="kube-rbac-proxy-metric" containerID="cri-o://a0875ea8e25b658a729adfc968fba494b811b65ce842f2e5a81f04bed34ec7e3" gracePeriod=120
Nov 01 05:46:45.133238 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:45.133213    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="prom-label-proxy" containerID="cri-o://a967a54f3d73124b9e832286f7d0f79b70882226af4361033325c327605c8de9" gracePeriod=120
Nov 01 05:46:45.133373 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:45.133231    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="kube-rbac-proxy" containerID="cri-o://b5e340a9ebe8aa6a315464a8436274221ecec244d501abf23ebb9f13bbdff343" gracePeriod=120
Nov 01 05:46:45.133373 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:45.133161    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="kube-rbac-proxy-web" containerID="cri-o://5b5625e8603c063020e3994de5783522f5ec8607b2fc6cc66144e97a4a62c179" gracePeriod=120
Nov 01 05:46:45.133373 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:45.133248    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="config-reloader" containerID="cri-o://ef6cd62ea245b7f75fb4bf3aa2d93e954ea422c80a6b066b48cbb0a6b0ca95bb" gracePeriod=120
Nov 01 05:46:46.236606 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:46.236576    5619 generic.go:356] "Generic (PLEG): container finished" podID="f4eea300-bfd9-465e-9276-16494acaa883" containerID="a967a54f3d73124b9e832286f7d0f79b70882226af4361033325c327605c8de9" exitCode=0
Nov 01 05:46:46.236606 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:46.236604    5619 generic.go:356] "Generic (PLEG): container finished" podID="f4eea300-bfd9-465e-9276-16494acaa883" containerID="a0875ea8e25b658a729adfc968fba494b811b65ce842f2e5a81f04bed34ec7e3" exitCode=0
Nov 01 05:46:46.236606 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:46.236610    5619 generic.go:356] "Generic (PLEG): container finished" podID="f4eea300-bfd9-465e-9276-16494acaa883" containerID="b5e340a9ebe8aa6a315464a8436274221ecec244d501abf23ebb9f13bbdff343" exitCode=0
Nov 01 05:46:46.237019 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:46.236617    5619 generic.go:356] "Generic (PLEG): container finished" podID="f4eea300-bfd9-465e-9276-16494acaa883" containerID="ef6cd62ea245b7f75fb4bf3aa2d93e954ea422c80a6b066b48cbb0a6b0ca95bb" exitCode=0
Nov 01 05:46:46.237019 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:46.236622    5619 generic.go:356] "Generic (PLEG): container finished" podID="f4eea300-bfd9-465e-9276-16494acaa883" containerID="6de0ae75117fddfa787c7ff5deb8d4ad1383a2937cdf7c5fc502728190ea31a1" exitCode=0
Nov 01 05:46:46.237019 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:46.236652    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f4eea300-bfd9-465e-9276-16494acaa883","Type":"ContainerDied","Data":"a967a54f3d73124b9e832286f7d0f79b70882226af4361033325c327605c8de9"}
Nov 01 05:46:46.237019 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:46.236738    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f4eea300-bfd9-465e-9276-16494acaa883","Type":"ContainerDied","Data":"a0875ea8e25b658a729adfc968fba494b811b65ce842f2e5a81f04bed34ec7e3"}
Nov 01 05:46:46.237019 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:46.236767    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f4eea300-bfd9-465e-9276-16494acaa883","Type":"ContainerDied","Data":"b5e340a9ebe8aa6a315464a8436274221ecec244d501abf23ebb9f13bbdff343"}
Nov 01 05:46:46.237019 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:46.236791    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f4eea300-bfd9-465e-9276-16494acaa883","Type":"ContainerDied","Data":"ef6cd62ea245b7f75fb4bf3aa2d93e954ea422c80a6b066b48cbb0a6b0ca95bb"}
Nov 01 05:46:46.237019 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:46.236814    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f4eea300-bfd9-465e-9276-16494acaa883","Type":"ContainerDied","Data":"6de0ae75117fddfa787c7ff5deb8d4ad1383a2937cdf7c5fc502728190ea31a1"}
Nov 01 05:46:46.903255 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:46.903228    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.076561 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.076483    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy-web\") pod \"f4eea300-bfd9-465e-9276-16494acaa883\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") "
Nov 01 05:46:47.076561 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.076525    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-main-tls\") pod \"f4eea300-bfd9-465e-9276-16494acaa883\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") "
Nov 01 05:46:47.076561 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.076549    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4eea300-bfd9-465e-9276-16494acaa883-tls-assets\" (UniqueName: \"kubernetes.io/projected/f4eea300-bfd9-465e-9276-16494acaa883-tls-assets\") pod \"f4eea300-bfd9-465e-9276-16494acaa883\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") "
Nov 01 05:46:47.076835 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.076581    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4eea300-bfd9-465e-9276-16494acaa883-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f4eea300-bfd9-465e-9276-16494acaa883-alertmanager-main-db\") pod \"f4eea300-bfd9-465e-9276-16494acaa883\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") "
Nov 01 05:46:47.076835 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.076628    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4eea300-bfd9-465e-9276-16494acaa883-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4eea300-bfd9-465e-9276-16494acaa883-metrics-client-ca\") pod \"f4eea300-bfd9-465e-9276-16494acaa883\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") "
Nov 01 05:46:47.076835 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.076656    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-config-volume\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-config-volume\") pod \"f4eea300-bfd9-465e-9276-16494acaa883\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") "
Nov 01 05:46:47.076835 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.076676    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4eea300-bfd9-465e-9276-16494acaa883-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4eea300-bfd9-465e-9276-16494acaa883-alertmanager-trusted-ca-bundle\") pod \"f4eea300-bfd9-465e-9276-16494acaa883\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") "
Nov 01 05:46:47.076835 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.076707    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-cluster-tls-config\") pod \"f4eea300-bfd9-465e-9276-16494acaa883\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") "
Nov 01 05:46:47.076835 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.076743    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4eea300-bfd9-465e-9276-16494acaa883-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f4eea300-bfd9-465e-9276-16494acaa883-config-out\") pod \"f4eea300-bfd9-465e-9276-16494acaa883\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") "
Nov 01 05:46:47.076835 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.076773    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-web-config\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-web-config\") pod \"f4eea300-bfd9-465e-9276-16494acaa883\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") "
Nov 01 05:46:47.076835 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.076810    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy\") pod \"f4eea300-bfd9-465e-9276-16494acaa883\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") "
Nov 01 05:46:47.077218 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.076862    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4eea300-bfd9-465e-9276-16494acaa883-kube-api-access-bkn77\" (UniqueName: \"kubernetes.io/projected/f4eea300-bfd9-465e-9276-16494acaa883-kube-api-access-bkn77\") pod \"f4eea300-bfd9-465e-9276-16494acaa883\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") "
Nov 01 05:46:47.077218 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.076860    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4eea300-bfd9-465e-9276-16494acaa883-alertmanager-main-db" pod "f4eea300-bfd9-465e-9276-16494acaa883" (UID: "f4eea300-bfd9-465e-9276-16494acaa883"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:46:47.077218 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.076904    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f4eea300-bfd9-465e-9276-16494acaa883\" (UID: \"f4eea300-bfd9-465e-9276-16494acaa883\") "
Nov 01 05:46:47.077218 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.077153    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4eea300-bfd9-465e-9276-16494acaa883-metrics-client-ca" pod "f4eea300-bfd9-465e-9276-16494acaa883" (UID: "f4eea300-bfd9-465e-9276-16494acaa883"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:46:47.077218 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.077162    5619 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f4eea300-bfd9-465e-9276-16494acaa883-alertmanager-main-db\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:47.078083 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.078045    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4eea300-bfd9-465e-9276-16494acaa883-alertmanager-trusted-ca-bundle" pod "f4eea300-bfd9-465e-9276-16494acaa883" (UID: "f4eea300-bfd9-465e-9276-16494acaa883"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:46:47.080465 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.080434    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy-metric" pod "f4eea300-bfd9-465e-9276-16494acaa883" (UID: "f4eea300-bfd9-465e-9276-16494acaa883"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:46:47.081048 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.080919    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy-web" pod "f4eea300-bfd9-465e-9276-16494acaa883" (UID: "f4eea300-bfd9-465e-9276-16494acaa883"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:46:47.081048 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.080972    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4eea300-bfd9-465e-9276-16494acaa883-kube-api-access-bkn77" pod "f4eea300-bfd9-465e-9276-16494acaa883" (UID: "f4eea300-bfd9-465e-9276-16494acaa883"). InnerVolumeSpecName "kube-api-access-bkn77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:46:47.081211 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.081160    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4eea300-bfd9-465e-9276-16494acaa883-config-out" pod "f4eea300-bfd9-465e-9276-16494acaa883" (UID: "f4eea300-bfd9-465e-9276-16494acaa883"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:46:47.081630 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.081607    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy" pod "f4eea300-bfd9-465e-9276-16494acaa883" (UID: "f4eea300-bfd9-465e-9276-16494acaa883"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:46:47.082360 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.082299    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-main-tls" pod "f4eea300-bfd9-465e-9276-16494acaa883" (UID: "f4eea300-bfd9-465e-9276-16494acaa883"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:46:47.082557 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.082528    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-config-volume" pod "f4eea300-bfd9-465e-9276-16494acaa883" (UID: "f4eea300-bfd9-465e-9276-16494acaa883"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:46:47.082557 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.082541    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4eea300-bfd9-465e-9276-16494acaa883-tls-assets" pod "f4eea300-bfd9-465e-9276-16494acaa883" (UID: "f4eea300-bfd9-465e-9276-16494acaa883"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:46:47.086384 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.086362    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-cluster-tls-config" pod "f4eea300-bfd9-465e-9276-16494acaa883" (UID: "f4eea300-bfd9-465e-9276-16494acaa883"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:46:47.091815 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.091790    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-web-config" pod "f4eea300-bfd9-465e-9276-16494acaa883" (UID: "f4eea300-bfd9-465e-9276-16494acaa883"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:46:47.178570 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.178543    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkn77\" (UniqueName: \"kubernetes.io/projected/f4eea300-bfd9-465e-9276-16494acaa883-kube-api-access-bkn77\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:47.178570 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.178567    5619 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:47.178713 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.178583    5619 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:47.178713 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.178598    5619 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-main-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:47.178713 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.178609    5619 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f4eea300-bfd9-465e-9276-16494acaa883-tls-assets\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:47.178713 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.178617    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4eea300-bfd9-465e-9276-16494acaa883-metrics-client-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:47.178713 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.178625    5619 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:47.178713 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.178634    5619 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4eea300-bfd9-465e-9276-16494acaa883-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:47.178713 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.178643    5619 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-cluster-tls-config\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:47.178713 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.178652    5619 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f4eea300-bfd9-465e-9276-16494acaa883-config-out\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:47.178713 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.178684    5619 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-web-config\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:47.178713 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.178693    5619 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4eea300-bfd9-465e-9276-16494acaa883-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:47.244560 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.244530    5619 generic.go:356] "Generic (PLEG): container finished" podID="f4eea300-bfd9-465e-9276-16494acaa883" containerID="5b5625e8603c063020e3994de5783522f5ec8607b2fc6cc66144e97a4a62c179" exitCode=0
Nov 01 05:46:47.244977 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.244567    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f4eea300-bfd9-465e-9276-16494acaa883","Type":"ContainerDied","Data":"5b5625e8603c063020e3994de5783522f5ec8607b2fc6cc66144e97a4a62c179"}
Nov 01 05:46:47.244977 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.244615    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.244977 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.244624    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f4eea300-bfd9-465e-9276-16494acaa883","Type":"ContainerDied","Data":"cc11cb67ba92c1e2a7c27ebe0b6e77b34723fa8217835ad395a71fb8fe3eee8a"}
Nov 01 05:46:47.244977 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.244660    5619 scope.go:122] "RemoveContainer" containerID="a967a54f3d73124b9e832286f7d0f79b70882226af4361033325c327605c8de9"
Nov 01 05:46:47.262639 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.262623    5619 scope.go:122] "RemoveContainer" containerID="a0875ea8e25b658a729adfc968fba494b811b65ce842f2e5a81f04bed34ec7e3"
Nov 01 05:46:47.267732 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.267709    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:47.269464 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.269443    5619 scope.go:122] "RemoveContainer" containerID="b5e340a9ebe8aa6a315464a8436274221ecec244d501abf23ebb9f13bbdff343"
Nov 01 05:46:47.275782 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.275764    5619 scope.go:122] "RemoveContainer" containerID="5b5625e8603c063020e3994de5783522f5ec8607b2fc6cc66144e97a4a62c179"
Nov 01 05:46:47.281936 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.281920    5619 scope.go:122] "RemoveContainer" containerID="ef6cd62ea245b7f75fb4bf3aa2d93e954ea422c80a6b066b48cbb0a6b0ca95bb"
Nov 01 05:46:47.288336 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.288310    5619 scope.go:122] "RemoveContainer" containerID="6de0ae75117fddfa787c7ff5deb8d4ad1383a2937cdf7c5fc502728190ea31a1"
Nov 01 05:46:47.294645 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.294631    5619 scope.go:122] "RemoveContainer" containerID="5f71a589277cb4dd289414e7212d5b8b60008db7afae72656b26c20e0134450a"
Nov 01 05:46:47.300778 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.300764    5619 scope.go:122] "RemoveContainer" containerID="a967a54f3d73124b9e832286f7d0f79b70882226af4361033325c327605c8de9"
Nov 01 05:46:47.301011 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:47.300994    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a967a54f3d73124b9e832286f7d0f79b70882226af4361033325c327605c8de9\": container with ID starting with a967a54f3d73124b9e832286f7d0f79b70882226af4361033325c327605c8de9 not found: ID does not exist" containerID="a967a54f3d73124b9e832286f7d0f79b70882226af4361033325c327605c8de9"
Nov 01 05:46:47.301069 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.301020    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a967a54f3d73124b9e832286f7d0f79b70882226af4361033325c327605c8de9"} err="failed to get container status \"a967a54f3d73124b9e832286f7d0f79b70882226af4361033325c327605c8de9\": rpc error: code = NotFound desc = could not find container \"a967a54f3d73124b9e832286f7d0f79b70882226af4361033325c327605c8de9\": container with ID starting with a967a54f3d73124b9e832286f7d0f79b70882226af4361033325c327605c8de9 not found: ID does not exist"
Nov 01 05:46:47.301069 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.301067    5619 scope.go:122] "RemoveContainer" containerID="a0875ea8e25b658a729adfc968fba494b811b65ce842f2e5a81f04bed34ec7e3"
Nov 01 05:46:47.301333 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:47.301303    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0875ea8e25b658a729adfc968fba494b811b65ce842f2e5a81f04bed34ec7e3\": container with ID starting with a0875ea8e25b658a729adfc968fba494b811b65ce842f2e5a81f04bed34ec7e3 not found: ID does not exist" containerID="a0875ea8e25b658a729adfc968fba494b811b65ce842f2e5a81f04bed34ec7e3"
Nov 01 05:46:47.301379 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.301346    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0875ea8e25b658a729adfc968fba494b811b65ce842f2e5a81f04bed34ec7e3"} err="failed to get container status \"a0875ea8e25b658a729adfc968fba494b811b65ce842f2e5a81f04bed34ec7e3\": rpc error: code = NotFound desc = could not find container \"a0875ea8e25b658a729adfc968fba494b811b65ce842f2e5a81f04bed34ec7e3\": container with ID starting with a0875ea8e25b658a729adfc968fba494b811b65ce842f2e5a81f04bed34ec7e3 not found: ID does not exist"
Nov 01 05:46:47.301379 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.301368    5619 scope.go:122] "RemoveContainer" containerID="b5e340a9ebe8aa6a315464a8436274221ecec244d501abf23ebb9f13bbdff343"
Nov 01 05:46:47.301644 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:47.301622    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e340a9ebe8aa6a315464a8436274221ecec244d501abf23ebb9f13bbdff343\": container with ID starting with b5e340a9ebe8aa6a315464a8436274221ecec244d501abf23ebb9f13bbdff343 not found: ID does not exist" containerID="b5e340a9ebe8aa6a315464a8436274221ecec244d501abf23ebb9f13bbdff343"
Nov 01 05:46:47.301709 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.301650    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e340a9ebe8aa6a315464a8436274221ecec244d501abf23ebb9f13bbdff343"} err="failed to get container status \"b5e340a9ebe8aa6a315464a8436274221ecec244d501abf23ebb9f13bbdff343\": rpc error: code = NotFound desc = could not find container \"b5e340a9ebe8aa6a315464a8436274221ecec244d501abf23ebb9f13bbdff343\": container with ID starting with b5e340a9ebe8aa6a315464a8436274221ecec244d501abf23ebb9f13bbdff343 not found: ID does not exist"
Nov 01 05:46:47.301709 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.301669    5619 scope.go:122] "RemoveContainer" containerID="5b5625e8603c063020e3994de5783522f5ec8607b2fc6cc66144e97a4a62c179"
Nov 01 05:46:47.301935 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:47.301916    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b5625e8603c063020e3994de5783522f5ec8607b2fc6cc66144e97a4a62c179\": container with ID starting with 5b5625e8603c063020e3994de5783522f5ec8607b2fc6cc66144e97a4a62c179 not found: ID does not exist" containerID="5b5625e8603c063020e3994de5783522f5ec8607b2fc6cc66144e97a4a62c179"
Nov 01 05:46:47.302006 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.301939    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b5625e8603c063020e3994de5783522f5ec8607b2fc6cc66144e97a4a62c179"} err="failed to get container status \"5b5625e8603c063020e3994de5783522f5ec8607b2fc6cc66144e97a4a62c179\": rpc error: code = NotFound desc = could not find container \"5b5625e8603c063020e3994de5783522f5ec8607b2fc6cc66144e97a4a62c179\": container with ID starting with 5b5625e8603c063020e3994de5783522f5ec8607b2fc6cc66144e97a4a62c179 not found: ID does not exist"
Nov 01 05:46:47.302006 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.301954    5619 scope.go:122] "RemoveContainer" containerID="ef6cd62ea245b7f75fb4bf3aa2d93e954ea422c80a6b066b48cbb0a6b0ca95bb"
Nov 01 05:46:47.302261 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:47.302228    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef6cd62ea245b7f75fb4bf3aa2d93e954ea422c80a6b066b48cbb0a6b0ca95bb\": container with ID starting with ef6cd62ea245b7f75fb4bf3aa2d93e954ea422c80a6b066b48cbb0a6b0ca95bb not found: ID does not exist" containerID="ef6cd62ea245b7f75fb4bf3aa2d93e954ea422c80a6b066b48cbb0a6b0ca95bb"
Nov 01 05:46:47.302356 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.302266    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef6cd62ea245b7f75fb4bf3aa2d93e954ea422c80a6b066b48cbb0a6b0ca95bb"} err="failed to get container status \"ef6cd62ea245b7f75fb4bf3aa2d93e954ea422c80a6b066b48cbb0a6b0ca95bb\": rpc error: code = NotFound desc = could not find container \"ef6cd62ea245b7f75fb4bf3aa2d93e954ea422c80a6b066b48cbb0a6b0ca95bb\": container with ID starting with ef6cd62ea245b7f75fb4bf3aa2d93e954ea422c80a6b066b48cbb0a6b0ca95bb not found: ID does not exist"
Nov 01 05:46:47.302356 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.302285    5619 scope.go:122] "RemoveContainer" containerID="6de0ae75117fddfa787c7ff5deb8d4ad1383a2937cdf7c5fc502728190ea31a1"
Nov 01 05:46:47.302615 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:47.302597    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6de0ae75117fddfa787c7ff5deb8d4ad1383a2937cdf7c5fc502728190ea31a1\": container with ID starting with 6de0ae75117fddfa787c7ff5deb8d4ad1383a2937cdf7c5fc502728190ea31a1 not found: ID does not exist" containerID="6de0ae75117fddfa787c7ff5deb8d4ad1383a2937cdf7c5fc502728190ea31a1"
Nov 01 05:46:47.302826 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.302620    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6de0ae75117fddfa787c7ff5deb8d4ad1383a2937cdf7c5fc502728190ea31a1"} err="failed to get container status \"6de0ae75117fddfa787c7ff5deb8d4ad1383a2937cdf7c5fc502728190ea31a1\": rpc error: code = NotFound desc = could not find container \"6de0ae75117fddfa787c7ff5deb8d4ad1383a2937cdf7c5fc502728190ea31a1\": container with ID starting with 6de0ae75117fddfa787c7ff5deb8d4ad1383a2937cdf7c5fc502728190ea31a1 not found: ID does not exist"
Nov 01 05:46:47.302826 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.302639    5619 scope.go:122] "RemoveContainer" containerID="5f71a589277cb4dd289414e7212d5b8b60008db7afae72656b26c20e0134450a"
Nov 01 05:46:47.303104 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:47.302902    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f71a589277cb4dd289414e7212d5b8b60008db7afae72656b26c20e0134450a\": container with ID starting with 5f71a589277cb4dd289414e7212d5b8b60008db7afae72656b26c20e0134450a not found: ID does not exist" containerID="5f71a589277cb4dd289414e7212d5b8b60008db7afae72656b26c20e0134450a"
Nov 01 05:46:47.303104 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.302920    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f71a589277cb4dd289414e7212d5b8b60008db7afae72656b26c20e0134450a"} err="failed to get container status \"5f71a589277cb4dd289414e7212d5b8b60008db7afae72656b26c20e0134450a\": rpc error: code = NotFound desc = could not find container \"5f71a589277cb4dd289414e7212d5b8b60008db7afae72656b26c20e0134450a\": container with ID starting with 5f71a589277cb4dd289414e7212d5b8b60008db7afae72656b26c20e0134450a not found: ID does not exist"
Nov 01 05:46:47.408538 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.408504    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:47.415552 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.415527    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 05:46:47.419147 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.419123    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 05:46:47.465226 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.465091    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 05:46:47.465437 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.465418    5619 cpu_manager.go:414] "containerMap: removing container" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="kube-rbac-proxy-web"
Nov 01 05:46:47.465524 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.465440    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="kube-rbac-proxy-web"
Nov 01 05:46:47.465524 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.465458    5619 cpu_manager.go:414] "containerMap: removing container" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="alertmanager"
Nov 01 05:46:47.465524 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.465466    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="alertmanager"
Nov 01 05:46:47.465524 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.465480    5619 cpu_manager.go:414] "containerMap: removing container" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="prom-label-proxy"
Nov 01 05:46:47.465524 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.465487    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="prom-label-proxy"
Nov 01 05:46:47.465524 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.465501    5619 cpu_manager.go:414] "containerMap: removing container" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="kube-rbac-proxy-metric"
Nov 01 05:46:47.465524 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.465508    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="kube-rbac-proxy-metric"
Nov 01 05:46:47.465524 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.465523    5619 cpu_manager.go:414] "containerMap: removing container" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="kube-rbac-proxy"
Nov 01 05:46:47.465782 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.465530    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="kube-rbac-proxy"
Nov 01 05:46:47.465782 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.465552    5619 cpu_manager.go:414] "containerMap: removing container" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="config-reloader"
Nov 01 05:46:47.465782 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.465559    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="config-reloader"
Nov 01 05:46:47.465782 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.465578    5619 cpu_manager.go:414] "containerMap: removing container" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="init-config-reloader"
Nov 01 05:46:47.465782 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.465585    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="init-config-reloader"
Nov 01 05:46:47.465782 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.465652    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="kube-rbac-proxy-metric"
Nov 01 05:46:47.465782 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.465665    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="kube-rbac-proxy"
Nov 01 05:46:47.465782 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.465676    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="config-reloader"
Nov 01 05:46:47.465782 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.465685    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="init-config-reloader"
Nov 01 05:46:47.465782 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.465695    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="kube-rbac-proxy-web"
Nov 01 05:46:47.465782 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.465702    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="alertmanager"
Nov 01 05:46:47.465782 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.465723    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4eea300-bfd9-465e-9276-16494acaa883" containerName="prom-label-proxy"
Nov 01 05:46:47.495924 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.495891    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.537571 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.537546    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 05:46:47.550090 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.550023    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 01 05:46:47.550090 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.550074    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 01 05:46:47.550090 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.550075    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 01 05:46:47.550597 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.550089    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 01 05:46:47.550597 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.550115    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 01 05:46:47.550597 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.550128    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 01 05:46:47.550597 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.550125    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 01 05:46:47.550597 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.550279    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 01 05:46:47.550597 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.550306    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-75cn8\""
Nov 01 05:46:47.587517 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.587498    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 01 05:46:47.683974 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.683911    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/63d18a9f-f582-4a00-a0b4-3d33f9363354-config-volume\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.683974 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.683942    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63d18a9f-f582-4a00-a0b4-3d33f9363354-web-config\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.683974 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.683968    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29ckg\" (UniqueName: \"kubernetes.io/projected/63d18a9f-f582-4a00-a0b4-3d33f9363354-kube-api-access-29ckg\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.684157 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.684120    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63d18a9f-f582-4a00-a0b4-3d33f9363354-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.684189 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.684153    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63d18a9f-f582-4a00-a0b4-3d33f9363354-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.684189 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.684184    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63d18a9f-f582-4a00-a0b4-3d33f9363354-tls-assets\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.684256 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.684239    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/63d18a9f-f582-4a00-a0b4-3d33f9363354-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.684307 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.684289    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/63d18a9f-f582-4a00-a0b4-3d33f9363354-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.684373 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.684340    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/63d18a9f-f582-4a00-a0b4-3d33f9363354-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.684475 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.684454    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63d18a9f-f582-4a00-a0b4-3d33f9363354-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.684534 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.684499    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63d18a9f-f582-4a00-a0b4-3d33f9363354-config-out\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.684585 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.684547    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63d18a9f-f582-4a00-a0b4-3d33f9363354-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.684644 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.684594    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/63d18a9f-f582-4a00-a0b4-3d33f9363354-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.786067 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.786038    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63d18a9f-f582-4a00-a0b4-3d33f9363354-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.786189 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.786072    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/63d18a9f-f582-4a00-a0b4-3d33f9363354-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.786189 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.786105    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/63d18a9f-f582-4a00-a0b4-3d33f9363354-config-volume\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.786189 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.786127    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63d18a9f-f582-4a00-a0b4-3d33f9363354-web-config\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.786189 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.786151    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29ckg\" (UniqueName: \"kubernetes.io/projected/63d18a9f-f582-4a00-a0b4-3d33f9363354-kube-api-access-29ckg\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.786439 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.786197    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63d18a9f-f582-4a00-a0b4-3d33f9363354-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.786439 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.786229    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63d18a9f-f582-4a00-a0b4-3d33f9363354-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.786439 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.786260    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63d18a9f-f582-4a00-a0b4-3d33f9363354-tls-assets\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.786439 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.786284    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/63d18a9f-f582-4a00-a0b4-3d33f9363354-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.786439 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.786344    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/63d18a9f-f582-4a00-a0b4-3d33f9363354-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.786439 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.786370    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/63d18a9f-f582-4a00-a0b4-3d33f9363354-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.786439 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.786418    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63d18a9f-f582-4a00-a0b4-3d33f9363354-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.786789 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.786455    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63d18a9f-f582-4a00-a0b4-3d33f9363354-config-out\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.786789 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.786510    5619 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 05:46:47.786907 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.786849    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63d18a9f-f582-4a00-a0b4-3d33f9363354-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.787106 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.787050    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/63d18a9f-f582-4a00-a0b4-3d33f9363354-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.787376 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.787293    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63d18a9f-f582-4a00-a0b4-3d33f9363354-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.789432 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.789407    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63d18a9f-f582-4a00-a0b4-3d33f9363354-config-out\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.789621 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.789595    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63d18a9f-f582-4a00-a0b4-3d33f9363354-tls-assets\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.789743 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.789694    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/63d18a9f-f582-4a00-a0b4-3d33f9363354-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.789810 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.789743    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63d18a9f-f582-4a00-a0b4-3d33f9363354-web-config\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.789864 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.789827    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/63d18a9f-f582-4a00-a0b4-3d33f9363354-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.790127 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.790104    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/63d18a9f-f582-4a00-a0b4-3d33f9363354-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.790198 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.790178    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63d18a9f-f582-4a00-a0b4-3d33f9363354-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.790238 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.790178    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/63d18a9f-f582-4a00-a0b4-3d33f9363354-config-volume\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.791346 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.791306    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63d18a9f-f582-4a00-a0b4-3d33f9363354-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.848595 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.848567    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29ckg\" (UniqueName: \"kubernetes.io/projected/63d18a9f-f582-4a00-a0b4-3d33f9363354-kube-api-access-29ckg\") pod \"alertmanager-main-0\" (UID: \"63d18a9f-f582-4a00-a0b4-3d33f9363354\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:47.898237 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:47.898214    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 05:46:48.356181 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:48.356155    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 05:46:48.359428 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:48.359389    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d18a9f_f582_4a00_a0b4_3d33f9363354.slice/crio-b2adfd0bdb35ba0ccbb2acf49a79b833ead1db80c6e6d091575e31c3a0d49aa8 WatchSource:0}: Error finding container b2adfd0bdb35ba0ccbb2acf49a79b833ead1db80c6e6d091575e31c3a0d49aa8: Status 404 returned error can't find the container with id b2adfd0bdb35ba0ccbb2acf49a79b833ead1db80c6e6d091575e31c3a0d49aa8
Nov 01 05:46:48.962868 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:48.962835    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4eea300-bfd9-465e-9276-16494acaa883" path="/var/lib/kubelet/pods/f4eea300-bfd9-465e-9276-16494acaa883/volumes"
Nov 01 05:46:49.252990 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:49.252954    5619 generic.go:356] "Generic (PLEG): container finished" podID="63d18a9f-f582-4a00-a0b4-3d33f9363354" containerID="ba2eff247d216df2e8ce1031ee724fb612b57cca02b27393cfdd3c01eb21f276" exitCode=0
Nov 01 05:46:49.253138 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:49.253050    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63d18a9f-f582-4a00-a0b4-3d33f9363354","Type":"ContainerDied","Data":"ba2eff247d216df2e8ce1031ee724fb612b57cca02b27393cfdd3c01eb21f276"}
Nov 01 05:46:49.253138 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:49.253131    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63d18a9f-f582-4a00-a0b4-3d33f9363354","Type":"ContainerStarted","Data":"b2adfd0bdb35ba0ccbb2acf49a79b833ead1db80c6e6d091575e31c3a0d49aa8"}
Nov 01 05:46:50.260264 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:50.260231    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63d18a9f-f582-4a00-a0b4-3d33f9363354","Type":"ContainerStarted","Data":"9e3c5b6af5f4f673a8a2dc15c21c46d8942771ab8eae7c0cb51be0e6b32f4a90"}
Nov 01 05:46:50.260719 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:50.260305    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63d18a9f-f582-4a00-a0b4-3d33f9363354","Type":"ContainerStarted","Data":"65183c935419ed5734e0331e2035094c91eb0e412417a727e248ec59d39d83e1"}
Nov 01 05:46:50.260719 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:50.260362    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63d18a9f-f582-4a00-a0b4-3d33f9363354","Type":"ContainerStarted","Data":"74e1b2f8ae63a58d23bc2e350dfae3c8cf42dbf59859e1e15b55d3a840ce8e1a"}
Nov 01 05:46:50.260719 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:50.260394    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63d18a9f-f582-4a00-a0b4-3d33f9363354","Type":"ContainerStarted","Data":"bbc2a41385b05a9920dce1091de5379bae5abc6ef90a548a9747535ccdf5fbec"}
Nov 01 05:46:50.260719 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:50.260418    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63d18a9f-f582-4a00-a0b4-3d33f9363354","Type":"ContainerStarted","Data":"251366d3bf8c8cd48778aef5adc8ca00c7ba4e898d8c1156f068b1149e350c64"}
Nov 01 05:46:50.260719 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:50.260450    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63d18a9f-f582-4a00-a0b4-3d33f9363354","Type":"ContainerStarted","Data":"4d96d808c4c7edf4fb73d54386a6d9306d9899188aa9624ae47be29c2b30fb00"}
Nov 01 05:46:50.392677 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:50.392630    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.379738158 podStartE2EDuration="3.392616851s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.878693ms" isStatefulPod=false podCreationTimestamp="2026-11-01 05:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:46:50.392150677 +0000 UTC m=+47.602974185" watchObservedRunningTime="2026-11-01 05:46:50.392616851 +0000 UTC m=+47.603440359"
Nov 01 05:46:50.821284 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:50.821251    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 05:46:50.821634 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:50.821600    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="prometheus" containerID="cri-o://2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79" gracePeriod=600
Nov 01 05:46:50.821792 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:50.821629    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="kube-rbac-proxy-web" containerID="cri-o://f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894" gracePeriod=600
Nov 01 05:46:50.821792 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:50.821658    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="kube-rbac-proxy-thanos" containerID="cri-o://366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7" gracePeriod=600
Nov 01 05:46:50.821792 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:50.821600    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="kube-rbac-proxy" containerID="cri-o://37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e" gracePeriod=600
Nov 01 05:46:50.821792 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:50.821628    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="thanos-sidecar" containerID="cri-o://db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a" gracePeriod=600
Nov 01 05:46:50.821792 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:50.821641    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="config-reloader" containerID="cri-o://81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40" gracePeriod=600
Nov 01 05:46:51.067745 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.067721    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.113507 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.113487    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-kube-rbac-proxy\") pod \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") "
Nov 01 05:46:51.113588 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.113520    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") "
Nov 01 05:46:51.113588 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.113542    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-metrics-client-certs\") pod \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") "
Nov 01 05:46:51.113710 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.113689    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-tls\") pod \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") "
Nov 01 05:46:51.113754 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.113743    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-db\") pod \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") "
Nov 01 05:46:51.113797 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.113776    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-kubelet-serving-ca-bundle\") pod \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") "
Nov 01 05:46:51.113853 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.113811    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-config-out\" (UniqueName: \"kubernetes.io/empty-dir/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-config-out\") pod \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") "
Nov 01 05:46:51.113910 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.113856    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-1\") pod \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") "
Nov 01 05:46:51.113910 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.113895    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-trusted-ca-bundle\") pod \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") "
Nov 01 05:46:51.114020 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.113939    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-grpc-tls\") pod \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") "
Nov 01 05:46:51.114020 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.113969    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-metrics-client-ca\") pod \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") "
Nov 01 05:46:51.114020 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.114012    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") "
Nov 01 05:46:51.114186 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.114046    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-0\") pod \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") "
Nov 01 05:46:51.114186 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.114081    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-tls-assets\" (UniqueName: \"kubernetes.io/projected/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-tls-assets\") pod \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") "
Nov 01 05:46:51.114186 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.114125    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-web-config\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-web-config\") pod \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") "
Nov 01 05:46:51.114186 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.114153    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-serving-certs-ca-bundle\") pod \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") "
Nov 01 05:46:51.114412 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.114188    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-thanos-prometheus-http-client-file\") pod \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") "
Nov 01 05:46:51.114412 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.114225    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-config\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-config\") pod \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") "
Nov 01 05:46:51.114412 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.114257    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-2\") pod \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") "
Nov 01 05:46:51.114412 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.114295    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-kube-api-access-rp4hf\" (UniqueName: \"kubernetes.io/projected/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-kube-api-access-rp4hf\") pod \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\" (UID: \"4254b02e-3f9a-4bcc-b711-de1bbb5b3395\") "
Nov 01 05:46:51.114412 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.114388    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-trusted-ca-bundle" pod "4254b02e-3f9a-4bcc-b711-de1bbb5b3395" (UID: "4254b02e-3f9a-4bcc-b711-de1bbb5b3395"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:46:51.114663 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.114583    5619 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:51.114663 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.114598    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-kubelet-serving-ca-bundle" pod "4254b02e-3f9a-4bcc-b711-de1bbb5b3395" (UID: "4254b02e-3f9a-4bcc-b711-de1bbb5b3395"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:46:51.114985 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.114852    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-db" pod "4254b02e-3f9a-4bcc-b711-de1bbb5b3395" (UID: "4254b02e-3f9a-4bcc-b711-de1bbb5b3395"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:46:51.115424 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.115397    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-metrics-client-ca" pod "4254b02e-3f9a-4bcc-b711-de1bbb5b3395" (UID: "4254b02e-3f9a-4bcc-b711-de1bbb5b3395"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:46:51.115424 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.115409    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-1" pod "4254b02e-3f9a-4bcc-b711-de1bbb5b3395" (UID: "4254b02e-3f9a-4bcc-b711-de1bbb5b3395"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:46:51.115800 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.115774    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-2" pod "4254b02e-3f9a-4bcc-b711-de1bbb5b3395" (UID: "4254b02e-3f9a-4bcc-b711-de1bbb5b3395"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:46:51.116141 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.116116    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-serving-certs-ca-bundle" pod "4254b02e-3f9a-4bcc-b711-de1bbb5b3395" (UID: "4254b02e-3f9a-4bcc-b711-de1bbb5b3395"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:46:51.116993 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.116965    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-tls" pod "4254b02e-3f9a-4bcc-b711-de1bbb5b3395" (UID: "4254b02e-3f9a-4bcc-b711-de1bbb5b3395"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:46:51.116993 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.116972    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-0" pod "4254b02e-3f9a-4bcc-b711-de1bbb5b3395" (UID: "4254b02e-3f9a-4bcc-b711-de1bbb5b3395"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:46:51.117151 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.117043    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-metrics-client-certs" pod "4254b02e-3f9a-4bcc-b711-de1bbb5b3395" (UID: "4254b02e-3f9a-4bcc-b711-de1bbb5b3395"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:46:51.117218 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.117151    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-kube-api-access-rp4hf" pod "4254b02e-3f9a-4bcc-b711-de1bbb5b3395" (UID: "4254b02e-3f9a-4bcc-b711-de1bbb5b3395"). InnerVolumeSpecName "kube-api-access-rp4hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:46:51.117281 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.117264    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-kube-rbac-proxy-web" pod "4254b02e-3f9a-4bcc-b711-de1bbb5b3395" (UID: "4254b02e-3f9a-4bcc-b711-de1bbb5b3395"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:46:51.117644 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.117608    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-kube-rbac-proxy" pod "4254b02e-3f9a-4bcc-b711-de1bbb5b3395" (UID: "4254b02e-3f9a-4bcc-b711-de1bbb5b3395"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:46:51.117876 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.117849    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-grpc-tls" pod "4254b02e-3f9a-4bcc-b711-de1bbb5b3395" (UID: "4254b02e-3f9a-4bcc-b711-de1bbb5b3395"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:46:51.118060 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.118041    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-tls-assets" pod "4254b02e-3f9a-4bcc-b711-de1bbb5b3395" (UID: "4254b02e-3f9a-4bcc-b711-de1bbb5b3395"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:46:51.118292 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.118275    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-thanos-prometheus-http-client-file" pod "4254b02e-3f9a-4bcc-b711-de1bbb5b3395" (UID: "4254b02e-3f9a-4bcc-b711-de1bbb5b3395"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:46:51.118377 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.118286    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-config-out" pod "4254b02e-3f9a-4bcc-b711-de1bbb5b3395" (UID: "4254b02e-3f9a-4bcc-b711-de1bbb5b3395"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 05:46:51.119062 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.119039    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-config" pod "4254b02e-3f9a-4bcc-b711-de1bbb5b3395" (UID: "4254b02e-3f9a-4bcc-b711-de1bbb5b3395"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:46:51.119196 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.119173    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-thanos-sidecar-tls" pod "4254b02e-3f9a-4bcc-b711-de1bbb5b3395" (UID: "4254b02e-3f9a-4bcc-b711-de1bbb5b3395"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:46:51.128111 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.128090    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-web-config" pod "4254b02e-3f9a-4bcc-b711-de1bbb5b3395" (UID: "4254b02e-3f9a-4bcc-b711-de1bbb5b3395"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:46:51.215073 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.215049    5619 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-metrics-client-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:51.215073 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.215074    5619 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:51.215215 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.215086    5619 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-db\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:51.215215 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.215097    5619 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:51.215215 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.215106    5619 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-config-out\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:51.215215 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.215115    5619 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:51.215215 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.215123    5619 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-grpc-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:51.215215 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.215132    5619 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-metrics-client-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:51.215215 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.215141    5619 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:51.215215 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.215151    5619 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:51.215215 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.215159    5619 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-tls-assets\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:51.215215 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.215168    5619 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-web-config\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:51.215215 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.215176    5619 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:51.215215 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.215185    5619 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:51.215215 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.215193    5619 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-config\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:51.215215 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.215202    5619 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:51.215215 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.215210    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp4hf\" (UniqueName: \"kubernetes.io/projected/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-kube-api-access-rp4hf\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:51.215215 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.215218    5619 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-kube-rbac-proxy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:51.215694 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.215227    5619 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4254b02e-3f9a-4bcc-b711-de1bbb5b3395-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:46:51.266859 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.266832    5619 generic.go:356] "Generic (PLEG): container finished" podID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerID="366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7" exitCode=0
Nov 01 05:46:51.266859 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.266858    5619 generic.go:356] "Generic (PLEG): container finished" podID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerID="37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e" exitCode=0
Nov 01 05:46:51.267283 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.266869    5619 generic.go:356] "Generic (PLEG): container finished" podID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerID="f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894" exitCode=0
Nov 01 05:46:51.267283 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.266877    5619 generic.go:356] "Generic (PLEG): container finished" podID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerID="db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a" exitCode=0
Nov 01 05:46:51.267283 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.266887    5619 generic.go:356] "Generic (PLEG): container finished" podID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerID="81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40" exitCode=0
Nov 01 05:46:51.267283 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.266895    5619 generic.go:356] "Generic (PLEG): container finished" podID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerID="2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79" exitCode=0
Nov 01 05:46:51.267283 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.266922    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.267283 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.266916    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4254b02e-3f9a-4bcc-b711-de1bbb5b3395","Type":"ContainerDied","Data":"366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7"}
Nov 01 05:46:51.267283 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.267067    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4254b02e-3f9a-4bcc-b711-de1bbb5b3395","Type":"ContainerDied","Data":"37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e"}
Nov 01 05:46:51.267283 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.267115    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4254b02e-3f9a-4bcc-b711-de1bbb5b3395","Type":"ContainerDied","Data":"f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894"}
Nov 01 05:46:51.267283 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.267133    5619 scope.go:122] "RemoveContainer" containerID="366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7"
Nov 01 05:46:51.267283 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.267164    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4254b02e-3f9a-4bcc-b711-de1bbb5b3395","Type":"ContainerDied","Data":"db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a"}
Nov 01 05:46:51.267283 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.267206    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4254b02e-3f9a-4bcc-b711-de1bbb5b3395","Type":"ContainerDied","Data":"81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40"}
Nov 01 05:46:51.267283 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.267249    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4254b02e-3f9a-4bcc-b711-de1bbb5b3395","Type":"ContainerDied","Data":"2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79"}
Nov 01 05:46:51.267283 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.267284    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4254b02e-3f9a-4bcc-b711-de1bbb5b3395","Type":"ContainerDied","Data":"8b45d9a43bd1cea46768ba85a72d27e5c6ee0b34f3b7d1db813b95d75a74c225"}
Nov 01 05:46:51.274288 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.274272    5619 scope.go:122] "RemoveContainer" containerID="37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e"
Nov 01 05:46:51.280937 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.280920    5619 scope.go:122] "RemoveContainer" containerID="f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894"
Nov 01 05:46:51.287030 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.287014    5619 scope.go:122] "RemoveContainer" containerID="db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a"
Nov 01 05:46:51.293121 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.293106    5619 scope.go:122] "RemoveContainer" containerID="81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40"
Nov 01 05:46:51.299162 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.299148    5619 scope.go:122] "RemoveContainer" containerID="2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79"
Nov 01 05:46:51.305803 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.305783    5619 scope.go:122] "RemoveContainer" containerID="0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03"
Nov 01 05:46:51.311867 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.311850    5619 scope.go:122] "RemoveContainer" containerID="366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7"
Nov 01 05:46:51.312121 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:51.312103    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7\": container with ID starting with 366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7 not found: ID does not exist" containerID="366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7"
Nov 01 05:46:51.312168 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.312128    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7"} err="failed to get container status \"366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7\": rpc error: code = NotFound desc = could not find container \"366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7\": container with ID starting with 366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7 not found: ID does not exist"
Nov 01 05:46:51.312168 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.312144    5619 scope.go:122] "RemoveContainer" containerID="37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e"
Nov 01 05:46:51.312372 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:51.312353    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e\": container with ID starting with 37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e not found: ID does not exist" containerID="37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e"
Nov 01 05:46:51.312409 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.312378    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e"} err="failed to get container status \"37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e\": rpc error: code = NotFound desc = could not find container \"37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e\": container with ID starting with 37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e not found: ID does not exist"
Nov 01 05:46:51.312409 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.312394    5619 scope.go:122] "RemoveContainer" containerID="f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894"
Nov 01 05:46:51.312634 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:51.312616    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894\": container with ID starting with f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894 not found: ID does not exist" containerID="f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894"
Nov 01 05:46:51.312698 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.312651    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894"} err="failed to get container status \"f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894\": rpc error: code = NotFound desc = could not find container \"f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894\": container with ID starting with f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894 not found: ID does not exist"
Nov 01 05:46:51.312698 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.312673    5619 scope.go:122] "RemoveContainer" containerID="db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a"
Nov 01 05:46:51.312892 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:51.312877    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a\": container with ID starting with db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a not found: ID does not exist" containerID="db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a"
Nov 01 05:46:51.312930 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.312895    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a"} err="failed to get container status \"db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a\": rpc error: code = NotFound desc = could not find container \"db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a\": container with ID starting with db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a not found: ID does not exist"
Nov 01 05:46:51.312930 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.312910    5619 scope.go:122] "RemoveContainer" containerID="81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40"
Nov 01 05:46:51.313137 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:51.313119    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40\": container with ID starting with 81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40 not found: ID does not exist" containerID="81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40"
Nov 01 05:46:51.313215 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.313146    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40"} err="failed to get container status \"81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40\": rpc error: code = NotFound desc = could not find container \"81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40\": container with ID starting with 81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40 not found: ID does not exist"
Nov 01 05:46:51.313215 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.313165    5619 scope.go:122] "RemoveContainer" containerID="2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79"
Nov 01 05:46:51.313461 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:51.313444    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79\": container with ID starting with 2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79 not found: ID does not exist" containerID="2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79"
Nov 01 05:46:51.313523 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.313465    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79"} err="failed to get container status \"2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79\": rpc error: code = NotFound desc = could not find container \"2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79\": container with ID starting with 2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79 not found: ID does not exist"
Nov 01 05:46:51.313523 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.313479    5619 scope.go:122] "RemoveContainer" containerID="0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03"
Nov 01 05:46:51.313686 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:51.313668    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03\": container with ID starting with 0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03 not found: ID does not exist" containerID="0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03"
Nov 01 05:46:51.313727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.313690    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03"} err="failed to get container status \"0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03\": rpc error: code = NotFound desc = could not find container \"0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03\": container with ID starting with 0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03 not found: ID does not exist"
Nov 01 05:46:51.313727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.313702    5619 scope.go:122] "RemoveContainer" containerID="366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7"
Nov 01 05:46:51.313920 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.313904    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7"} err="failed to get container status \"366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7\": rpc error: code = NotFound desc = could not find container \"366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7\": container with ID starting with 366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7 not found: ID does not exist"
Nov 01 05:46:51.313968 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.313920    5619 scope.go:122] "RemoveContainer" containerID="37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e"
Nov 01 05:46:51.314121 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.314106    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e"} err="failed to get container status \"37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e\": rpc error: code = NotFound desc = could not find container \"37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e\": container with ID starting with 37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e not found: ID does not exist"
Nov 01 05:46:51.314173 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.314121    5619 scope.go:122] "RemoveContainer" containerID="f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894"
Nov 01 05:46:51.314353 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.314299    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894"} err="failed to get container status \"f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894\": rpc error: code = NotFound desc = could not find container \"f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894\": container with ID starting with f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894 not found: ID does not exist"
Nov 01 05:46:51.314404 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.314354    5619 scope.go:122] "RemoveContainer" containerID="db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a"
Nov 01 05:46:51.314609 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.314591    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a"} err="failed to get container status \"db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a\": rpc error: code = NotFound desc = could not find container \"db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a\": container with ID starting with db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a not found: ID does not exist"
Nov 01 05:46:51.314666 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.314610    5619 scope.go:122] "RemoveContainer" containerID="81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40"
Nov 01 05:46:51.314861 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.314841    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40"} err="failed to get container status \"81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40\": rpc error: code = NotFound desc = could not find container \"81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40\": container with ID starting with 81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40 not found: ID does not exist"
Nov 01 05:46:51.314910 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.314862    5619 scope.go:122] "RemoveContainer" containerID="2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79"
Nov 01 05:46:51.315081 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.315064    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79"} err="failed to get container status \"2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79\": rpc error: code = NotFound desc = could not find container \"2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79\": container with ID starting with 2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79 not found: ID does not exist"
Nov 01 05:46:51.315116 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.315081    5619 scope.go:122] "RemoveContainer" containerID="0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03"
Nov 01 05:46:51.315315 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.315296    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03"} err="failed to get container status \"0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03\": rpc error: code = NotFound desc = could not find container \"0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03\": container with ID starting with 0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03 not found: ID does not exist"
Nov 01 05:46:51.315361 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.315317    5619 scope.go:122] "RemoveContainer" containerID="366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7"
Nov 01 05:46:51.315647 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.315623    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7"} err="failed to get container status \"366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7\": rpc error: code = NotFound desc = could not find container \"366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7\": container with ID starting with 366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7 not found: ID does not exist"
Nov 01 05:46:51.315647 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.315645    5619 scope.go:122] "RemoveContainer" containerID="37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e"
Nov 01 05:46:51.316190 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.316167    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e"} err="failed to get container status \"37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e\": rpc error: code = NotFound desc = could not find container \"37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e\": container with ID starting with 37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e not found: ID does not exist"
Nov 01 05:46:51.316271 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.316192    5619 scope.go:122] "RemoveContainer" containerID="f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894"
Nov 01 05:46:51.316459 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.316442    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894"} err="failed to get container status \"f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894\": rpc error: code = NotFound desc = could not find container \"f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894\": container with ID starting with f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894 not found: ID does not exist"
Nov 01 05:46:51.316534 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.316461    5619 scope.go:122] "RemoveContainer" containerID="db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a"
Nov 01 05:46:51.316678 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.316660    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a"} err="failed to get container status \"db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a\": rpc error: code = NotFound desc = could not find container \"db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a\": container with ID starting with db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a not found: ID does not exist"
Nov 01 05:46:51.316721 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.316679    5619 scope.go:122] "RemoveContainer" containerID="81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40"
Nov 01 05:46:51.316883 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.316867    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40"} err="failed to get container status \"81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40\": rpc error: code = NotFound desc = could not find container \"81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40\": container with ID starting with 81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40 not found: ID does not exist"
Nov 01 05:46:51.316948 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.316884    5619 scope.go:122] "RemoveContainer" containerID="2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79"
Nov 01 05:46:51.317088 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.317072    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79"} err="failed to get container status \"2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79\": rpc error: code = NotFound desc = could not find container \"2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79\": container with ID starting with 2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79 not found: ID does not exist"
Nov 01 05:46:51.317138 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.317088    5619 scope.go:122] "RemoveContainer" containerID="0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03"
Nov 01 05:46:51.317314 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.317296    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03"} err="failed to get container status \"0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03\": rpc error: code = NotFound desc = could not find container \"0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03\": container with ID starting with 0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03 not found: ID does not exist"
Nov 01 05:46:51.317383 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.317314    5619 scope.go:122] "RemoveContainer" containerID="366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7"
Nov 01 05:46:51.317534 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.317518    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7"} err="failed to get container status \"366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7\": rpc error: code = NotFound desc = could not find container \"366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7\": container with ID starting with 366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7 not found: ID does not exist"
Nov 01 05:46:51.317576 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.317534    5619 scope.go:122] "RemoveContainer" containerID="37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e"
Nov 01 05:46:51.317746 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.317732    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e"} err="failed to get container status \"37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e\": rpc error: code = NotFound desc = could not find container \"37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e\": container with ID starting with 37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e not found: ID does not exist"
Nov 01 05:46:51.317786 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.317746    5619 scope.go:122] "RemoveContainer" containerID="f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894"
Nov 01 05:46:51.317947 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.317931    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894"} err="failed to get container status \"f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894\": rpc error: code = NotFound desc = could not find container \"f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894\": container with ID starting with f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894 not found: ID does not exist"
Nov 01 05:46:51.317947 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.317946    5619 scope.go:122] "RemoveContainer" containerID="db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a"
Nov 01 05:46:51.318142 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.318128    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a"} err="failed to get container status \"db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a\": rpc error: code = NotFound desc = could not find container \"db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a\": container with ID starting with db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a not found: ID does not exist"
Nov 01 05:46:51.318187 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.318142    5619 scope.go:122] "RemoveContainer" containerID="81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40"
Nov 01 05:46:51.318352 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.318335    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40"} err="failed to get container status \"81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40\": rpc error: code = NotFound desc = could not find container \"81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40\": container with ID starting with 81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40 not found: ID does not exist"
Nov 01 05:46:51.318390 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.318351    5619 scope.go:122] "RemoveContainer" containerID="2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79"
Nov 01 05:46:51.318585 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.318571    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79"} err="failed to get container status \"2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79\": rpc error: code = NotFound desc = could not find container \"2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79\": container with ID starting with 2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79 not found: ID does not exist"
Nov 01 05:46:51.318585 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.318583    5619 scope.go:122] "RemoveContainer" containerID="0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03"
Nov 01 05:46:51.318790 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.318774    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03"} err="failed to get container status \"0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03\": rpc error: code = NotFound desc = could not find container \"0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03\": container with ID starting with 0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03 not found: ID does not exist"
Nov 01 05:46:51.318844 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.318790    5619 scope.go:122] "RemoveContainer" containerID="366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7"
Nov 01 05:46:51.318994 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.318976    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7"} err="failed to get container status \"366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7\": rpc error: code = NotFound desc = could not find container \"366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7\": container with ID starting with 366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7 not found: ID does not exist"
Nov 01 05:46:51.318994 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.318992    5619 scope.go:122] "RemoveContainer" containerID="37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e"
Nov 01 05:46:51.319179 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.319164    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e"} err="failed to get container status \"37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e\": rpc error: code = NotFound desc = could not find container \"37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e\": container with ID starting with 37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e not found: ID does not exist"
Nov 01 05:46:51.319179 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.319179    5619 scope.go:122] "RemoveContainer" containerID="f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894"
Nov 01 05:46:51.319404 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.319383    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894"} err="failed to get container status \"f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894\": rpc error: code = NotFound desc = could not find container \"f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894\": container with ID starting with f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894 not found: ID does not exist"
Nov 01 05:46:51.319404 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.319403    5619 scope.go:122] "RemoveContainer" containerID="db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a"
Nov 01 05:46:51.319636 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.319618    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a"} err="failed to get container status \"db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a\": rpc error: code = NotFound desc = could not find container \"db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a\": container with ID starting with db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a not found: ID does not exist"
Nov 01 05:46:51.319694 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.319638    5619 scope.go:122] "RemoveContainer" containerID="81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40"
Nov 01 05:46:51.319845 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.319827    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40"} err="failed to get container status \"81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40\": rpc error: code = NotFound desc = could not find container \"81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40\": container with ID starting with 81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40 not found: ID does not exist"
Nov 01 05:46:51.319901 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.319845    5619 scope.go:122] "RemoveContainer" containerID="2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79"
Nov 01 05:46:51.320069 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.320049    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79"} err="failed to get container status \"2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79\": rpc error: code = NotFound desc = could not find container \"2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79\": container with ID starting with 2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79 not found: ID does not exist"
Nov 01 05:46:51.320121 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.320070    5619 scope.go:122] "RemoveContainer" containerID="0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03"
Nov 01 05:46:51.320332 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.320303    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03"} err="failed to get container status \"0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03\": rpc error: code = NotFound desc = could not find container \"0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03\": container with ID starting with 0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03 not found: ID does not exist"
Nov 01 05:46:51.320384 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.320341    5619 scope.go:122] "RemoveContainer" containerID="366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7"
Nov 01 05:46:51.320567 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.320550    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7"} err="failed to get container status \"366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7\": rpc error: code = NotFound desc = could not find container \"366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7\": container with ID starting with 366387f416a2bcd881bd0370cdadb048327e1836495992cefb8e59f4f6df3ad7 not found: ID does not exist"
Nov 01 05:46:51.320612 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.320567    5619 scope.go:122] "RemoveContainer" containerID="37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e"
Nov 01 05:46:51.320958 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.320931    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e"} err="failed to get container status \"37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e\": rpc error: code = NotFound desc = could not find container \"37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e\": container with ID starting with 37c68f0eeed9959ba10964f233740b76b2d25f663ed39f70612b48ceda25c92e not found: ID does not exist"
Nov 01 05:46:51.320958 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.320955    5619 scope.go:122] "RemoveContainer" containerID="f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894"
Nov 01 05:46:51.321213 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.321193    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894"} err="failed to get container status \"f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894\": rpc error: code = NotFound desc = could not find container \"f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894\": container with ID starting with f6c05b1182bf5b8c2aafd1144f8c2e84a041c52b6372ea85c9ca81102df25894 not found: ID does not exist"
Nov 01 05:46:51.321261 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.321213    5619 scope.go:122] "RemoveContainer" containerID="db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a"
Nov 01 05:46:51.321456 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.321438    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a"} err="failed to get container status \"db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a\": rpc error: code = NotFound desc = could not find container \"db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a\": container with ID starting with db7925991370d30a5626072c97316884b25738dcceb8afa1097e88ff0da51d2a not found: ID does not exist"
Nov 01 05:46:51.321504 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.321458    5619 scope.go:122] "RemoveContainer" containerID="81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40"
Nov 01 05:46:51.321694 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.321674    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40"} err="failed to get container status \"81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40\": rpc error: code = NotFound desc = could not find container \"81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40\": container with ID starting with 81416eb2d30a7e27e900b90d9766aa51aac0a493478a6ec92ac2d1a84dc0fd40 not found: ID does not exist"
Nov 01 05:46:51.321739 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.321696    5619 scope.go:122] "RemoveContainer" containerID="2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79"
Nov 01 05:46:51.321932 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.321912    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79"} err="failed to get container status \"2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79\": rpc error: code = NotFound desc = could not find container \"2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79\": container with ID starting with 2f0d6e02abe977a61e891834027fd0a9cf6b5a485c73b93d73dc60aafa7c7a79 not found: ID does not exist"
Nov 01 05:46:51.321975 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.321933    5619 scope.go:122] "RemoveContainer" containerID="0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03"
Nov 01 05:46:51.322190 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.322162    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03"} err="failed to get container status \"0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03\": rpc error: code = NotFound desc = could not find container \"0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03\": container with ID starting with 0ea65797f7b96ab707dbf1a5b9970fa321197da3db86fa5369ed9b8619858a03 not found: ID does not exist"
Nov 01 05:46:51.439649 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.439587    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 05:46:51.444145 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.444126    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 05:46:51.475572 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.475553    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 05:46:51.475804 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.475793    5619 cpu_manager.go:414] "containerMap: removing container" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="prometheus"
Nov 01 05:46:51.475837 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.475805    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="prometheus"
Nov 01 05:46:51.475837 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.475827    5619 cpu_manager.go:414] "containerMap: removing container" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="init-config-reloader"
Nov 01 05:46:51.475837 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.475836    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="init-config-reloader"
Nov 01 05:46:51.475933 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.475849    5619 cpu_manager.go:414] "containerMap: removing container" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="kube-rbac-proxy-thanos"
Nov 01 05:46:51.475933 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.475854    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="kube-rbac-proxy-thanos"
Nov 01 05:46:51.475933 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.475865    5619 cpu_manager.go:414] "containerMap: removing container" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="kube-rbac-proxy-web"
Nov 01 05:46:51.475933 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.475870    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="kube-rbac-proxy-web"
Nov 01 05:46:51.475933 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.475876    5619 cpu_manager.go:414] "containerMap: removing container" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="kube-rbac-proxy"
Nov 01 05:46:51.475933 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.475881    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="kube-rbac-proxy"
Nov 01 05:46:51.475933 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.475889    5619 cpu_manager.go:414] "containerMap: removing container" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="config-reloader"
Nov 01 05:46:51.475933 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.475893    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="config-reloader"
Nov 01 05:46:51.475933 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.475912    5619 cpu_manager.go:414] "containerMap: removing container" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="thanos-sidecar"
Nov 01 05:46:51.475933 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.475917    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="thanos-sidecar"
Nov 01 05:46:51.476202 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.475960    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="prometheus"
Nov 01 05:46:51.476202 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.475969    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="kube-rbac-proxy-thanos"
Nov 01 05:46:51.476202 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.475975    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="kube-rbac-proxy-web"
Nov 01 05:46:51.476202 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.475983    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="kube-rbac-proxy"
Nov 01 05:46:51.476202 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.475991    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="config-reloader"
Nov 01 05:46:51.476202 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.475998    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" containerName="thanos-sidecar"
Nov 01 05:46:51.479609 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.479595    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.517476 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.517446    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.517604 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.517488    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj7kv\" (UniqueName: \"kubernetes.io/projected/0d57ece9-d021-4395-b469-86a4efa4d857-kube-api-access-pj7kv\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.517604 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.517517    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0d57ece9-d021-4395-b469-86a4efa4d857-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.517604 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.517580    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d57ece9-d021-4395-b469-86a4efa4d857-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.517765 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.517616    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0d57ece9-d021-4395-b469-86a4efa4d857-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.517765 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.517707    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.517765 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.517741    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0d57ece9-d021-4395-b469-86a4efa4d857-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.517888 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.517773    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-config\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.517888 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.517810    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.517993 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.517906    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d57ece9-d021-4395-b469-86a4efa4d857-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.517993 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.517950    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d57ece9-d021-4395-b469-86a4efa4d857-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.517993 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.517981    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d57ece9-d021-4395-b469-86a4efa4d857-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.518103 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.518006    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.518103 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.518072    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.518103 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.518097    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0d57ece9-d021-4395-b469-86a4efa4d857-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.518217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.518128    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d57ece9-d021-4395-b469-86a4efa4d857-config-out\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.518217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.518144    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d57ece9-d021-4395-b469-86a4efa4d857-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.518217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.518195    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.518347 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.518228    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.518347 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.518258    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-web-config\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.533344 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.533315    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 01 05:46:51.533437 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.533339    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 01 05:46:51.533437 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.533342    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 01 05:46:51.533962 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.533939    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 01 05:46:51.534069 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.534010    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 01 05:46:51.534145 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.534118    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 01 05:46:51.534264 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.534245    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 01 05:46:51.534364 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.534274    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vdjzw\""
Nov 01 05:46:51.534364 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.534279    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 01 05:46:51.534364 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.534306    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 01 05:46:51.534364 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.534294    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 01 05:46:51.534585 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.534375    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 01 05:46:51.534664 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.534650    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 01 05:46:51.534742 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.534717    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dmpoc73dr8cgo\""
Nov 01 05:46:51.585029 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.585000    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 01 05:46:51.585132 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.585097    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 01 05:46:51.585984 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.585967    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 05:46:51.618863 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.618841    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0d57ece9-d021-4395-b469-86a4efa4d857-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.618972 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.618870    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d57ece9-d021-4395-b469-86a4efa4d857-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.618972 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.618886    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0d57ece9-d021-4395-b469-86a4efa4d857-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.618972 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.618909    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.618972 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.618936    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0d57ece9-d021-4395-b469-86a4efa4d857-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.618972 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.618964    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-config\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.619180 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.619087    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.619180 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.619141    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d57ece9-d021-4395-b469-86a4efa4d857-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.619254 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.619182    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d57ece9-d021-4395-b469-86a4efa4d857-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.619254 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.619205    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d57ece9-d021-4395-b469-86a4efa4d857-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.619254 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.619229    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.619421 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.619264    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.619421 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.619288    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0d57ece9-d021-4395-b469-86a4efa4d857-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.619421 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.619309    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0d57ece9-d021-4395-b469-86a4efa4d857-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.619421 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.619353    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d57ece9-d021-4395-b469-86a4efa4d857-config-out\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.619421 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.619379    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d57ece9-d021-4395-b469-86a4efa4d857-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.619421 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.619409    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.619715 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.619443    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.619715 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.619466    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-web-config\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.619715 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.619511    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.619715 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.619544    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj7kv\" (UniqueName: \"kubernetes.io/projected/0d57ece9-d021-4395-b469-86a4efa4d857-kube-api-access-pj7kv\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.619926 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.619740    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0d57ece9-d021-4395-b469-86a4efa4d857-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.620661 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.620238    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d57ece9-d021-4395-b469-86a4efa4d857-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.620661 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.620469    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d57ece9-d021-4395-b469-86a4efa4d857-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.622502 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.620871    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d57ece9-d021-4395-b469-86a4efa4d857-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.622502 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.621411    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0d57ece9-d021-4395-b469-86a4efa4d857-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.622502 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.621440    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d57ece9-d021-4395-b469-86a4efa4d857-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.622502 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.622280    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-config\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.622502 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.622362    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.623183 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.623161    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0d57ece9-d021-4395-b469-86a4efa4d857-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.623314 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.623221    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.624186 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.623940    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.624186 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.623979    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d57ece9-d021-4395-b469-86a4efa4d857-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.624186 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.624148    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.624409 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.624278    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.624515 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.624490    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.624756 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.624736    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.625439 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.625423    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d57ece9-d021-4395-b469-86a4efa4d857-web-config\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.625533 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.625472    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d57ece9-d021-4395-b469-86a4efa4d857-config-out\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.681367 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.681316    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj7kv\" (UniqueName: \"kubernetes.io/projected/0d57ece9-d021-4395-b469-86a4efa4d857-kube-api-access-pj7kv\") pod \"prometheus-k8s-0\" (UID: \"0d57ece9-d021-4395-b469-86a4efa4d857\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:51.796033 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:51.795970    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:46:52.228451 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:52.227917    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 05:46:52.230267 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:46:52.230239    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d57ece9_d021_4395_b469_86a4efa4d857.slice/crio-31be79562a88d82e8168a134458c19f35db36a647b44ea046c3a4a27999b2d9f WatchSource:0}: Error finding container 31be79562a88d82e8168a134458c19f35db36a647b44ea046c3a4a27999b2d9f: Status 404 returned error can't find the container with id 31be79562a88d82e8168a134458c19f35db36a647b44ea046c3a4a27999b2d9f
Nov 01 05:46:52.271378 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:52.271348    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d57ece9-d021-4395-b469-86a4efa4d857","Type":"ContainerStarted","Data":"31be79562a88d82e8168a134458c19f35db36a647b44ea046c3a4a27999b2d9f"}
Nov 01 05:46:52.959870 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:52.959835    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4254b02e-3f9a-4bcc-b711-de1bbb5b3395" path="/var/lib/kubelet/pods/4254b02e-3f9a-4bcc-b711-de1bbb5b3395/volumes"
Nov 01 05:46:53.277212 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:53.277128    5619 generic.go:356] "Generic (PLEG): container finished" podID="0d57ece9-d021-4395-b469-86a4efa4d857" containerID="b2f80daa4af414c57658336a26c3829197af3fe5d932b2ce0258a0f9d1de359e" exitCode=0
Nov 01 05:46:53.277593 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:53.277230    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d57ece9-d021-4395-b469-86a4efa4d857","Type":"ContainerDied","Data":"b2f80daa4af414c57658336a26c3829197af3fe5d932b2ce0258a0f9d1de359e"}
Nov 01 05:46:54.284941 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:54.284905    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d57ece9-d021-4395-b469-86a4efa4d857","Type":"ContainerStarted","Data":"d0afc4fbc59d8c883cde2295481953d901b2eddbdc5eba2e4bed2f339647c4bf"}
Nov 01 05:46:54.285285 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:54.284982    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d57ece9-d021-4395-b469-86a4efa4d857","Type":"ContainerStarted","Data":"9d173809224782a1338087731439c81c7179bb0fe52bb6e80d99fdd5a1ef90d6"}
Nov 01 05:46:54.285285 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:54.285019    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d57ece9-d021-4395-b469-86a4efa4d857","Type":"ContainerStarted","Data":"7e1ad0f87e00668e81eec5b57d7b28766c6af0cbe73e709611ae0d4406848256"}
Nov 01 05:46:54.285285 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:54.285059    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d57ece9-d021-4395-b469-86a4efa4d857","Type":"ContainerStarted","Data":"f7a924afa0f412d7c0cfe9fd4bd325289d1d52c4bb429bfa46ea29f8f7335db7"}
Nov 01 05:46:54.285285 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:54.285094    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d57ece9-d021-4395-b469-86a4efa4d857","Type":"ContainerStarted","Data":"68db03bfbe08c18365cb544673b6050a9622e38c6e32b6ef21e9ddcddff05dba"}
Nov 01 05:46:54.285285 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:54.285149    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0d57ece9-d021-4395-b469-86a4efa4d857","Type":"ContainerStarted","Data":"4d13a407fad4998cf5077a8b0fadac7603017a34dbb969c437232a2287107c50"}
Nov 01 05:46:54.428714 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:54.428661    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.415728025 podStartE2EDuration="3.428649772s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.921747ms" isStatefulPod=false podCreationTimestamp="2026-11-01 05:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:46:54.423989165 +0000 UTC m=+51.634812674" watchObservedRunningTime="2026-11-01 05:46:54.428649772 +0000 UTC m=+51.639473281"
Nov 01 05:46:55.360598 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:55.360559    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:55.499168 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:55.499135    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:55.762081 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:55.761997    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:55.877476 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:46:55.877438    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:46:55.904400 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:55.904369    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:46:56.796342 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:46:56.796281    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:47:03.781531 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:47:03.781494    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:47:03.796772 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:47:03.796748    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:47:04.333984 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:47:04.333961    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 05:47:10.780133 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:47:10.780096    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:10.921039 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:47:10.921010    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:11.416764 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:47:11.416734    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:15.752634 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:47:15.752596    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:15.894054 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:47:15.894025    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:16.159537 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:47:16.159499    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:16.299432 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:47:16.299400    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:23.325955 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:47:23.325922    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:23.540365 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:47:23.540318    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:25.877842 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:47:25.877810    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:47:27.733645 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:47:27.733608    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:28.559766 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:47:28.559723    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:39.712864 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:47:39.712829    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:39.861409 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:47:39.861370    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:55.763292 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:47:55.763208    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:47:55.878153 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:47:55.878109    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:48:11.670197 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:48:11.670167    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:11.817734 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:48:11.817702    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:25.877547 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:48:25.877502    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:48:34.541550 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:48:34.541513    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:50.860515 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:48:50.860478    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:51.051689 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:48:51.051654    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:48:55.878727 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:48:55.878691    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:49:12.125906 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:49:12.125822    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:12.320651 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:49:12.320611    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:49:25.880747 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:49:25.880707    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:49:55.878684 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:49:55.878649    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:50:12.475556 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:12.475519    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:25.516170 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:25.516133    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:25.760783 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:25.760752    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:25.878382 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:50:25.878343    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:50:25.909861 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:25.909834    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:27.090258 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:27.090218    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:27.280921 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:27.280888    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:28.340317 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:28.340279    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:34.057431 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:34.057399    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:34.196594 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:34.196556    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:35.421430 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.421396    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:35.628466 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.628431    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:35.641890 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.641862    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"]
Nov 01 05:50:35.643776 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.643760    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.686382 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.685250    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsv8d\" (UniqueName: \"kubernetes.io/projected/45899a85-a566-4cff-be6e-a2b0d6c7afed-kube-api-access-nsv8d\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.686382 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.685301    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45899a85-a566-4cff-be6e-a2b0d6c7afed-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.686382 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.685361    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45899a85-a566-4cff-be6e-a2b0d6c7afed-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.686382 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.685395    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45899a85-a566-4cff-be6e-a2b0d6c7afed-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.686382 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.685422    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45899a85-a566-4cff-be6e-a2b0d6c7afed-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.686382 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.685550    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45899a85-a566-4cff-be6e-a2b0d6c7afed-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.686382 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.685580    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45899a85-a566-4cff-be6e-a2b0d6c7afed-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.686382 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.685597    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45899a85-a566-4cff-be6e-a2b0d6c7afed-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.686382 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.685612    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45899a85-a566-4cff-be6e-a2b0d6c7afed-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.697287 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.697267    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 05:50:35.736152 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.736122    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-rkh9n\""
Nov 01 05:50:35.736152 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.736117    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 05:50:35.736152 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.736132    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 05:50:35.748791 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.748771    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"]
Nov 01 05:50:35.768247 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.768214    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:35.786363 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.786343    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsv8d\" (UniqueName: \"kubernetes.io/projected/45899a85-a566-4cff-be6e-a2b0d6c7afed-kube-api-access-nsv8d\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.786445 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.786373    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45899a85-a566-4cff-be6e-a2b0d6c7afed-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.786505 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.786490    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45899a85-a566-4cff-be6e-a2b0d6c7afed-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.786558 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.786519    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45899a85-a566-4cff-be6e-a2b0d6c7afed-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.786558 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.786534    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45899a85-a566-4cff-be6e-a2b0d6c7afed-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.786657 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.786564    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45899a85-a566-4cff-be6e-a2b0d6c7afed-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.786657 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.786595    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45899a85-a566-4cff-be6e-a2b0d6c7afed-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.786750 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.786642    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45899a85-a566-4cff-be6e-a2b0d6c7afed-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.786750 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.786683    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45899a85-a566-4cff-be6e-a2b0d6c7afed-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.786885 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.786867    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45899a85-a566-4cff-be6e-a2b0d6c7afed-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.787234 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.787211    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45899a85-a566-4cff-be6e-a2b0d6c7afed-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.787356 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.787244    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45899a85-a566-4cff-be6e-a2b0d6c7afed-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.787356 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.787279    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45899a85-a566-4cff-be6e-a2b0d6c7afed-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.787502 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.787485    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45899a85-a566-4cff-be6e-a2b0d6c7afed-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.788857 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.788831    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45899a85-a566-4cff-be6e-a2b0d6c7afed-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.788947 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.788932    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45899a85-a566-4cff-be6e-a2b0d6c7afed-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.847744 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.847720    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsv8d\" (UniqueName: \"kubernetes.io/projected/45899a85-a566-4cff-be6e-a2b0d6c7afed-kube-api-access-nsv8d\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.850252 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.850220    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45899a85-a566-4cff-be6e-a2b0d6c7afed-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl\" (UID: \"45899a85-a566-4cff-be6e-a2b0d6c7afed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:35.954814 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:35.954756    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:36.377024 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:36.376879    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"]
Nov 01 05:50:36.379515 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:50:36.379489    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45899a85_a566_4cff_be6e_a2b0d6c7afed.slice/crio-9b83ee9d4fa8591f3496663001ecd5e0ee31d37dea07acb455bd603a65624993 WatchSource:0}: Error finding container 9b83ee9d4fa8591f3496663001ecd5e0ee31d37dea07acb455bd603a65624993: Status 404 returned error can't find the container with id 9b83ee9d4fa8591f3496663001ecd5e0ee31d37dea07acb455bd603a65624993
Nov 01 05:50:37.015265 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:37.015236    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl" event={"ID":"45899a85-a566-4cff-be6e-a2b0d6c7afed","Type":"ContainerStarted","Data":"9b83ee9d4fa8591f3496663001ecd5e0ee31d37dea07acb455bd603a65624993"}
Nov 01 05:50:39.590581 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:39.590544    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:50:40.475916 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:40.475882    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:50:40.476038 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:40.475965    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:50:40.476038 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:40.476007    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:50:41.029548 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:41.029508    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl" event={"ID":"45899a85-a566-4cff-be6e-a2b0d6c7afed","Type":"ContainerStarted","Data":"588554dbeb66f7dcf8136f4deab0ff2ef35e9bb8dfeeb5d2212efe513027846d"}
Nov 01 05:50:41.202189 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:41.202138    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl" podStartSLOduration=2.108485534 podStartE2EDuration="6.202124222s" totalImagesPullingTime="4.093638688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:50:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:50:41.201480437 +0000 UTC m=+278.412303945" watchObservedRunningTime="2026-11-01 05:50:41.202124222 +0000 UTC m=+278.412947727"
Nov 01 05:50:41.955122 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:41.955091    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:41.959533 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:41.959509    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:42.032965 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:42.032943    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:42.033732 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:50:42.033714    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl"
Nov 01 05:50:55.877508 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:50:55.877469    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:51:07.046143 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:07.046094    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:07.195895 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:07.195862    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:07.663938 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:07.663900    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:07.923279 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:07.923195    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"]
Nov 01 05:51:07.926518 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:07.926500    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"
Nov 01 05:51:07.955161 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:07.955133    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:08.044690 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.044666    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 05:51:08.044690 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.044680    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 01 05:51:08.044877 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.044687    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 05:51:08.044877 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.044687    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 05:51:08.044877 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.044736    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-cq826\""
Nov 01 05:51:08.044877 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.044761    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 05:51:08.051133 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.051104    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f13c4757-62fd-430f-b50f-87f002d5ce57-apiservice-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-4bslc\" (UID: \"f13c4757-62fd-430f-b50f-87f002d5ce57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"
Nov 01 05:51:08.051467 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.051143    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f13c4757-62fd-430f-b50f-87f002d5ce57-webhook-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-4bslc\" (UID: \"f13c4757-62fd-430f-b50f-87f002d5ce57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"
Nov 01 05:51:08.051467 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.051175    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f13c4757-62fd-430f-b50f-87f002d5ce57-metrics-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-4bslc\" (UID: \"f13c4757-62fd-430f-b50f-87f002d5ce57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"
Nov 01 05:51:08.051467 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.051294    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f13c4757-62fd-430f-b50f-87f002d5ce57-manifests\") pod \"opendatahub-operator-controller-manager-985c476f9-4bslc\" (UID: \"f13c4757-62fd-430f-b50f-87f002d5ce57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"
Nov 01 05:51:08.051467 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.051317    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f13c4757-62fd-430f-b50f-87f002d5ce57-tmp\") pod \"opendatahub-operator-controller-manager-985c476f9-4bslc\" (UID: \"f13c4757-62fd-430f-b50f-87f002d5ce57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"
Nov 01 05:51:08.051467 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.051359    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc654\" (UniqueName: \"kubernetes.io/projected/f13c4757-62fd-430f-b50f-87f002d5ce57-kube-api-access-cc654\") pod \"opendatahub-operator-controller-manager-985c476f9-4bslc\" (UID: \"f13c4757-62fd-430f-b50f-87f002d5ce57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"
Nov 01 05:51:08.108986 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.108960    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:08.152030 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.152006    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f13c4757-62fd-430f-b50f-87f002d5ce57-manifests\") pod \"opendatahub-operator-controller-manager-985c476f9-4bslc\" (UID: \"f13c4757-62fd-430f-b50f-87f002d5ce57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"
Nov 01 05:51:08.152146 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.152032    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f13c4757-62fd-430f-b50f-87f002d5ce57-tmp\") pod \"opendatahub-operator-controller-manager-985c476f9-4bslc\" (UID: \"f13c4757-62fd-430f-b50f-87f002d5ce57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"
Nov 01 05:51:08.152146 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.152058    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc654\" (UniqueName: \"kubernetes.io/projected/f13c4757-62fd-430f-b50f-87f002d5ce57-kube-api-access-cc654\") pod \"opendatahub-operator-controller-manager-985c476f9-4bslc\" (UID: \"f13c4757-62fd-430f-b50f-87f002d5ce57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"
Nov 01 05:51:08.152146 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.152087    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f13c4757-62fd-430f-b50f-87f002d5ce57-apiservice-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-4bslc\" (UID: \"f13c4757-62fd-430f-b50f-87f002d5ce57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"
Nov 01 05:51:08.152146 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.152113    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f13c4757-62fd-430f-b50f-87f002d5ce57-webhook-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-4bslc\" (UID: \"f13c4757-62fd-430f-b50f-87f002d5ce57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"
Nov 01 05:51:08.152371 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.152150    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f13c4757-62fd-430f-b50f-87f002d5ce57-metrics-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-4bslc\" (UID: \"f13c4757-62fd-430f-b50f-87f002d5ce57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"
Nov 01 05:51:08.152528 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.152508    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f13c4757-62fd-430f-b50f-87f002d5ce57-manifests\") pod \"opendatahub-operator-controller-manager-985c476f9-4bslc\" (UID: \"f13c4757-62fd-430f-b50f-87f002d5ce57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"
Nov 01 05:51:08.154458 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.154430    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f13c4757-62fd-430f-b50f-87f002d5ce57-tmp\") pod \"opendatahub-operator-controller-manager-985c476f9-4bslc\" (UID: \"f13c4757-62fd-430f-b50f-87f002d5ce57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"
Nov 01 05:51:08.154565 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.154475    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f13c4757-62fd-430f-b50f-87f002d5ce57-metrics-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-4bslc\" (UID: \"f13c4757-62fd-430f-b50f-87f002d5ce57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"
Nov 01 05:51:08.154624 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.154584    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f13c4757-62fd-430f-b50f-87f002d5ce57-webhook-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-4bslc\" (UID: \"f13c4757-62fd-430f-b50f-87f002d5ce57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"
Nov 01 05:51:08.154680 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.154629    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f13c4757-62fd-430f-b50f-87f002d5ce57-apiservice-cert\") pod \"opendatahub-operator-controller-manager-985c476f9-4bslc\" (UID: \"f13c4757-62fd-430f-b50f-87f002d5ce57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"
Nov 01 05:51:08.240156 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.240098    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc654\" (UniqueName: \"kubernetes.io/projected/f13c4757-62fd-430f-b50f-87f002d5ce57-kube-api-access-cc654\") pod \"opendatahub-operator-controller-manager-985c476f9-4bslc\" (UID: \"f13c4757-62fd-430f-b50f-87f002d5ce57\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"
Nov 01 05:51:08.240284 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.240234    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"
Nov 01 05:51:08.261224 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.261191    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:08.277914 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.277890    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"]
Nov 01 05:51:08.472280 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.472251    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:08.936924 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.936895    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"]
Nov 01 05:51:08.938352 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:51:08.938316    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf13c4757_62fd_430f_b50f_87f002d5ce57.slice/crio-4a322bc8758a68e8f052c9e552c656a9de789f152164a129b475a8638011b2a9 WatchSource:0}: Error finding container 4a322bc8758a68e8f052c9e552c656a9de789f152164a129b475a8638011b2a9: Status 404 returned error can't find the container with id 4a322bc8758a68e8f052c9e552c656a9de789f152164a129b475a8638011b2a9
Nov 01 05:51:08.940074 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:08.940051    5619 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 05:51:09.125801 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:09.125767    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc" event={"ID":"f13c4757-62fd-430f-b50f-87f002d5ce57","Type":"ContainerStarted","Data":"4a322bc8758a68e8f052c9e552c656a9de789f152164a129b475a8638011b2a9"}
Nov 01 05:51:10.879011 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:10.878964    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:11.031601 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:11.031565    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:11.412990 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:11.412949    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:11.618846 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:11.618792    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:12.173932 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:12.173891    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:12.836143 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:12.836110    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:13.142942 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:13.142910    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc" event={"ID":"f13c4757-62fd-430f-b50f-87f002d5ce57","Type":"ContainerStarted","Data":"8515e75524bb9f129ce40211f467d59a265049fd5b2883cbc3695be03b4fe723"}
Nov 01 05:51:13.731746 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:13.731713    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:13.928835 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:13.928796    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:14.147332 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:14.147285    5619 generic.go:356] "Generic (PLEG): container finished" podID="f13c4757-62fd-430f-b50f-87f002d5ce57" containerID="8515e75524bb9f129ce40211f467d59a265049fd5b2883cbc3695be03b4fe723" exitCode=0
Nov 01 05:51:14.147498 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:14.147373    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc" event={"ID":"f13c4757-62fd-430f-b50f-87f002d5ce57","Type":"ContainerDied","Data":"8515e75524bb9f129ce40211f467d59a265049fd5b2883cbc3695be03b4fe723"}
Nov 01 05:51:15.153751 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:15.153716    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc" event={"ID":"f13c4757-62fd-430f-b50f-87f002d5ce57","Type":"ContainerStarted","Data":"218d3b7f670ecb8c611989a5b72423a17be53500f09c51a28ba2f9fda19af5e7"}
Nov 01 05:51:15.154107 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:15.153825    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"
Nov 01 05:51:15.188863 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:15.188831    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:15.334860 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:15.334774    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:15.392695 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:15.392655    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc" podStartSLOduration=3.428352634 podStartE2EDuration="8.392642668s" totalImagesPullingTime="4.55175363s" totalInitContainerRuntime="412.536404ms" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:15.341428664 +0000 UTC m=+312.552252173" watchObservedRunningTime="2026-11-01 05:51:15.392642668 +0000 UTC m=+312.603466170"
Nov 01 05:51:16.769835 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:16.769805    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:17.564206 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:17.564169    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:18.313438 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:18.313403    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-4ql2n"]
Nov 01 05:51:18.319690 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:18.319672    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4ql2n"
Nov 01 05:51:18.431297 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:18.431265    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 01 05:51:18.431489 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:18.431425    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 01 05:51:18.431555 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:18.431527    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-h75nn\""
Nov 01 05:51:18.432356 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:18.432313    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-4ql2n"]
Nov 01 05:51:18.439624 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:18.439605    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fdde11e-e193-4f42-86ad-35b15a6fa7fb-tmp\") pod \"jobset-operator-674cd59dc5-4ql2n\" (UID: \"0fdde11e-e193-4f42-86ad-35b15a6fa7fb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4ql2n"
Nov 01 05:51:18.439684 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:18.439642    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d88w6\" (UniqueName: \"kubernetes.io/projected/0fdde11e-e193-4f42-86ad-35b15a6fa7fb-kube-api-access-d88w6\") pod \"jobset-operator-674cd59dc5-4ql2n\" (UID: \"0fdde11e-e193-4f42-86ad-35b15a6fa7fb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4ql2n"
Nov 01 05:51:18.540389 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:18.540356    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fdde11e-e193-4f42-86ad-35b15a6fa7fb-tmp\") pod \"jobset-operator-674cd59dc5-4ql2n\" (UID: \"0fdde11e-e193-4f42-86ad-35b15a6fa7fb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4ql2n"
Nov 01 05:51:18.540549 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:18.540403    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d88w6\" (UniqueName: \"kubernetes.io/projected/0fdde11e-e193-4f42-86ad-35b15a6fa7fb-kube-api-access-d88w6\") pod \"jobset-operator-674cd59dc5-4ql2n\" (UID: \"0fdde11e-e193-4f42-86ad-35b15a6fa7fb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4ql2n"
Nov 01 05:51:18.540741 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:18.540721    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fdde11e-e193-4f42-86ad-35b15a6fa7fb-tmp\") pod \"jobset-operator-674cd59dc5-4ql2n\" (UID: \"0fdde11e-e193-4f42-86ad-35b15a6fa7fb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4ql2n"
Nov 01 05:51:18.604906 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:18.604879    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d88w6\" (UniqueName: \"kubernetes.io/projected/0fdde11e-e193-4f42-86ad-35b15a6fa7fb-kube-api-access-d88w6\") pod \"jobset-operator-674cd59dc5-4ql2n\" (UID: \"0fdde11e-e193-4f42-86ad-35b15a6fa7fb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4ql2n"
Nov 01 05:51:18.606142 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:18.606110    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:18.645038 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:18.645007    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4ql2n"
Nov 01 05:51:19.019783 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:19.019702    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-4ql2n"]
Nov 01 05:51:19.023111 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:51:19.023079    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fdde11e_e193_4f42_86ad_35b15a6fa7fb.slice/crio-6fe2627129c0fee564ab8eb7441e1fccf746eed3613d6f691c841d69643aea15 WatchSource:0}: Error finding container 6fe2627129c0fee564ab8eb7441e1fccf746eed3613d6f691c841d69643aea15: Status 404 returned error can't find the container with id 6fe2627129c0fee564ab8eb7441e1fccf746eed3613d6f691c841d69643aea15
Nov 01 05:51:19.170155 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:19.170124    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4ql2n" event={"ID":"0fdde11e-e193-4f42-86ad-35b15a6fa7fb","Type":"ContainerStarted","Data":"6fe2627129c0fee564ab8eb7441e1fccf746eed3613d6f691c841d69643aea15"}
Nov 01 05:51:22.773946 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:22.773911    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:23.054776 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:23.054706    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:24.192494 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:24.192460    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4ql2n" event={"ID":"0fdde11e-e193-4f42-86ad-35b15a6fa7fb","Type":"ContainerStarted","Data":"6a78c67b5bb805a80f64c7e03655f29d046f4a60f3c0de5a7ba73f0a15ef4cb1"}
Nov 01 05:51:24.310484 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:24.310435    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4ql2n" podStartSLOduration=1.760009376 podStartE2EDuration="6.310419505s" totalImagesPullingTime="4.550410129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:24.309836132 +0000 UTC m=+321.520659642" watchObservedRunningTime="2026-11-01 05:51:24.310419505 +0000 UTC m=+321.521243014"
Nov 01 05:51:25.582170 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:25.582133    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:25.877983 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:51:25.877940    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:51:26.159770 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:26.159697    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-985c476f9-4bslc"
Nov 01 05:51:26.599094 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:26.599062    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:27.165201 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:27.165166    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:27.970453 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:27.970417    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:30.814544 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:30.814509    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:32.154675 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:32.154638    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:33.298893 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:33.298852    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:33.447818 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:33.447789    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q"]
Nov 01 05:51:33.449762 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:33.449726    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:33.451214 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:33.451198    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q"
Nov 01 05:51:33.505954 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:33.505921    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q"]
Nov 01 05:51:33.547861 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:33.547832    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3857be3-6a36-43f7-adec-7c924b444dd1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zxf6q\" (UID: \"b3857be3-6a36-43f7-adec-7c924b444dd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q"
Nov 01 05:51:33.547978 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:33.547884    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvmr4\" (UniqueName: \"kubernetes.io/projected/b3857be3-6a36-43f7-adec-7c924b444dd1-kube-api-access-mvmr4\") pod \"kuadrant-operator-controller-manager-678d87766f-zxf6q\" (UID: \"b3857be3-6a36-43f7-adec-7c924b444dd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q"
Nov 01 05:51:33.555683 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:33.555622    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vhvbn\""
Nov 01 05:51:33.555683 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:33.555633    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 05:51:33.555683 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:33.555634    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 05:51:33.648238 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:33.648212    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3857be3-6a36-43f7-adec-7c924b444dd1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zxf6q\" (UID: \"b3857be3-6a36-43f7-adec-7c924b444dd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q"
Nov 01 05:51:33.648346 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:33.648254    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvmr4\" (UniqueName: \"kubernetes.io/projected/b3857be3-6a36-43f7-adec-7c924b444dd1-kube-api-access-mvmr4\") pod \"kuadrant-operator-controller-manager-678d87766f-zxf6q\" (UID: \"b3857be3-6a36-43f7-adec-7c924b444dd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q"
Nov 01 05:51:33.648612 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:33.648595    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3857be3-6a36-43f7-adec-7c924b444dd1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zxf6q\" (UID: \"b3857be3-6a36-43f7-adec-7c924b444dd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q"
Nov 01 05:51:33.709944 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:33.709923    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvmr4\" (UniqueName: \"kubernetes.io/projected/b3857be3-6a36-43f7-adec-7c924b444dd1-kube-api-access-mvmr4\") pod \"kuadrant-operator-controller-manager-678d87766f-zxf6q\" (UID: \"b3857be3-6a36-43f7-adec-7c924b444dd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q"
Nov 01 05:51:33.763048 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:33.763027    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q"
Nov 01 05:51:34.238125 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:34.238096    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q"]
Nov 01 05:51:34.241474 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:51:34.241444    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3857be3_6a36_43f7_adec_7c924b444dd1.slice/crio-24f7672e0e7e55007db4d97c89159a83fbbbac0b1ab664538816cd5dd553b89b WatchSource:0}: Error finding container 24f7672e0e7e55007db4d97c89159a83fbbbac0b1ab664538816cd5dd553b89b: Status 404 returned error can't find the container with id 24f7672e0e7e55007db4d97c89159a83fbbbac0b1ab664538816cd5dd553b89b
Nov 01 05:51:35.232404 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:35.232369    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q" event={"ID":"b3857be3-6a36-43f7-adec-7c924b444dd1","Type":"ContainerStarted","Data":"24f7672e0e7e55007db4d97c89159a83fbbbac0b1ab664538816cd5dd553b89b"}
Nov 01 05:51:35.265469 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:35.265437    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:35.506182 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:35.506092    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:35.693831 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:35.693797    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:37.565577 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:37.565540    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:38.035118 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:38.035087    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:39.983932 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:39.983890    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:40.172073 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:40.172033    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:41.584716 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:41.584682    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:41.774954 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:41.774921    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:42.266386 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:42.266318    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q" event={"ID":"b3857be3-6a36-43f7-adec-7c924b444dd1","Type":"ContainerStarted","Data":"ef8e560d23db763b749620081c987357bb4d4c8af10f60e29bfa64112e0a7990"}
Nov 01 05:51:42.266576 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:42.266535    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q"
Nov 01 05:51:42.388331 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:42.388272    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q" podStartSLOduration=1.963663645 podStartE2EDuration="9.388258177s" totalImagesPullingTime="7.424594532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:42.387009398 +0000 UTC m=+339.597832907" watchObservedRunningTime="2026-11-01 05:51:42.388258177 +0000 UTC m=+339.599081683"
Nov 01 05:51:42.768294 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:42.768255    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:42.958689 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:42.958654    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:43.932541 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:43.932510    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:44.122018 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:44.121985    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:45.740930 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:45.740886    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:46.035410 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:46.035301    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:46.228660 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:46.228613    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:46.769733 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:46.769695    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:46.959385 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:46.959355    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:47.753538 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:47.753506    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-zt4cd"]
Nov 01 05:51:47.758969 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:47.758948    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-zt4cd"
Nov 01 05:51:47.865250 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:47.865209    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3f69acb-521f-416f-9054-7fdf4ff8646a-bound-sa-token\") pod \"cert-manager-597b95cb84-zt4cd\" (UID: \"c3f69acb-521f-416f-9054-7fdf4ff8646a\") " pod="cert-manager/cert-manager-597b95cb84-zt4cd"
Nov 01 05:51:47.865250 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:47.865225    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 01 05:51:47.865685 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:47.865297    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6nv9\" (UniqueName: \"kubernetes.io/projected/c3f69acb-521f-416f-9054-7fdf4ff8646a-kube-api-access-l6nv9\") pod \"cert-manager-597b95cb84-zt4cd\" (UID: \"c3f69acb-521f-416f-9054-7fdf4ff8646a\") " pod="cert-manager/cert-manager-597b95cb84-zt4cd"
Nov 01 05:51:47.865685 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:47.865464    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 01 05:51:47.865685 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:47.865518    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-2vmgq\""
Nov 01 05:51:47.866437 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:47.866421    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-zt4cd"]
Nov 01 05:51:47.966057 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:47.966024    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3f69acb-521f-416f-9054-7fdf4ff8646a-bound-sa-token\") pod \"cert-manager-597b95cb84-zt4cd\" (UID: \"c3f69acb-521f-416f-9054-7fdf4ff8646a\") " pod="cert-manager/cert-manager-597b95cb84-zt4cd"
Nov 01 05:51:47.966231 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:47.966065    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6nv9\" (UniqueName: \"kubernetes.io/projected/c3f69acb-521f-416f-9054-7fdf4ff8646a-kube-api-access-l6nv9\") pod \"cert-manager-597b95cb84-zt4cd\" (UID: \"c3f69acb-521f-416f-9054-7fdf4ff8646a\") " pod="cert-manager/cert-manager-597b95cb84-zt4cd"
Nov 01 05:51:48.034865 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:48.034798    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3f69acb-521f-416f-9054-7fdf4ff8646a-bound-sa-token\") pod \"cert-manager-597b95cb84-zt4cd\" (UID: \"c3f69acb-521f-416f-9054-7fdf4ff8646a\") " pod="cert-manager/cert-manager-597b95cb84-zt4cd"
Nov 01 05:51:48.034865 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:48.034837    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6nv9\" (UniqueName: \"kubernetes.io/projected/c3f69acb-521f-416f-9054-7fdf4ff8646a-kube-api-access-l6nv9\") pod \"cert-manager-597b95cb84-zt4cd\" (UID: \"c3f69acb-521f-416f-9054-7fdf4ff8646a\") " pod="cert-manager/cert-manager-597b95cb84-zt4cd"
Nov 01 05:51:48.068964 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:48.068937    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-zt4cd"
Nov 01 05:51:48.105803 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:48.105766    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:48.302894 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:48.302816    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:48.440925 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:48.440895    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-zt4cd"]
Nov 01 05:51:48.446200 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:51:48.446170    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3f69acb_521f_416f_9054_7fdf4ff8646a.slice/crio-d9a4a7e0aeb49cbc17cecc0e8366dfbc8e5d7f015ad4329c9c1ae1e87f030a15 WatchSource:0}: Error finding container d9a4a7e0aeb49cbc17cecc0e8366dfbc8e5d7f015ad4329c9c1ae1e87f030a15: Status 404 returned error can't find the container with id d9a4a7e0aeb49cbc17cecc0e8366dfbc8e5d7f015ad4329c9c1ae1e87f030a15
Nov 01 05:51:48.714622 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:48.714587    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:48.864919 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:48.864891    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:49.294689 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:49.294652    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-zt4cd" event={"ID":"c3f69acb-521f-416f-9054-7fdf4ff8646a","Type":"ContainerStarted","Data":"d9a4a7e0aeb49cbc17cecc0e8366dfbc8e5d7f015ad4329c9c1ae1e87f030a15"}
Nov 01 05:51:50.153087 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.153047    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:50.232442 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.232409    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-84b94dcd67-zxvcc"]
Nov 01 05:51:50.234656 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.234634    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84b94dcd67-zxvcc"
Nov 01 05:51:50.284115 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.284080    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8797e16-282e-476e-8008-92456f432069-tmp\") pod \"kube-auth-proxy-84b94dcd67-zxvcc\" (UID: \"e8797e16-282e-476e-8008-92456f432069\") " pod="openshift-ingress/kube-auth-proxy-84b94dcd67-zxvcc"
Nov 01 05:51:50.284227 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.284196    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8797e16-282e-476e-8008-92456f432069-tls-certs\") pod \"kube-auth-proxy-84b94dcd67-zxvcc\" (UID: \"e8797e16-282e-476e-8008-92456f432069\") " pod="openshift-ingress/kube-auth-proxy-84b94dcd67-zxvcc"
Nov 01 05:51:50.284302 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.284253    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk4t9\" (UniqueName: \"kubernetes.io/projected/e8797e16-282e-476e-8008-92456f432069-kube-api-access-mk4t9\") pod \"kube-auth-proxy-84b94dcd67-zxvcc\" (UID: \"e8797e16-282e-476e-8008-92456f432069\") " pod="openshift-ingress/kube-auth-proxy-84b94dcd67-zxvcc"
Nov 01 05:51:50.287867 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.287843    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 01 05:51:50.288056 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.288039    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 01 05:51:50.338830 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.338799    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fqlfv\""
Nov 01 05:51:50.340500 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.340475    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84b94dcd67-zxvcc"]
Nov 01 05:51:50.385651 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.385600    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8797e16-282e-476e-8008-92456f432069-tls-certs\") pod \"kube-auth-proxy-84b94dcd67-zxvcc\" (UID: \"e8797e16-282e-476e-8008-92456f432069\") " pod="openshift-ingress/kube-auth-proxy-84b94dcd67-zxvcc"
Nov 01 05:51:50.385790 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.385658    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk4t9\" (UniqueName: \"kubernetes.io/projected/e8797e16-282e-476e-8008-92456f432069-kube-api-access-mk4t9\") pod \"kube-auth-proxy-84b94dcd67-zxvcc\" (UID: \"e8797e16-282e-476e-8008-92456f432069\") " pod="openshift-ingress/kube-auth-proxy-84b94dcd67-zxvcc"
Nov 01 05:51:50.385790 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.385705    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8797e16-282e-476e-8008-92456f432069-tmp\") pod \"kube-auth-proxy-84b94dcd67-zxvcc\" (UID: \"e8797e16-282e-476e-8008-92456f432069\") " pod="openshift-ingress/kube-auth-proxy-84b94dcd67-zxvcc"
Nov 01 05:51:50.388717 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.388690    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8797e16-282e-476e-8008-92456f432069-tmp\") pod \"kube-auth-proxy-84b94dcd67-zxvcc\" (UID: \"e8797e16-282e-476e-8008-92456f432069\") " pod="openshift-ingress/kube-auth-proxy-84b94dcd67-zxvcc"
Nov 01 05:51:50.388858 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.388841    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8797e16-282e-476e-8008-92456f432069-tls-certs\") pod \"kube-auth-proxy-84b94dcd67-zxvcc\" (UID: \"e8797e16-282e-476e-8008-92456f432069\") " pod="openshift-ingress/kube-auth-proxy-84b94dcd67-zxvcc"
Nov 01 05:51:50.436794 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.436669    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:50.451029 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.451005    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk4t9\" (UniqueName: \"kubernetes.io/projected/e8797e16-282e-476e-8008-92456f432069-kube-api-access-mk4t9\") pod \"kube-auth-proxy-84b94dcd67-zxvcc\" (UID: \"e8797e16-282e-476e-8008-92456f432069\") " pod="openshift-ingress/kube-auth-proxy-84b94dcd67-zxvcc"
Nov 01 05:51:50.549186 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.549152    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84b94dcd67-zxvcc"
Nov 01 05:51:50.755165 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.755090    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"]
Nov 01 05:51:50.757766 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.757732    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:50.758019 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.758001    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.820449 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.820417    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"]
Nov 01 05:51:50.827520 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.827490    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-wxpv9\""
Nov 01 05:51:50.891512 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.891476    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c04bed-9587-4bd7-904c-cc0846d94e86-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.891694 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.891546    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c04bed-9587-4bd7-904c-cc0846d94e86-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.891694 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.891669    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c04bed-9587-4bd7-904c-cc0846d94e86-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.891807 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.891705    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c04bed-9587-4bd7-904c-cc0846d94e86-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.891807 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.891752    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r7fw\" (UniqueName: \"kubernetes.io/projected/41c04bed-9587-4bd7-904c-cc0846d94e86-kube-api-access-5r7fw\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.891807 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.891796    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c04bed-9587-4bd7-904c-cc0846d94e86-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.891955 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.891815    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c04bed-9587-4bd7-904c-cc0846d94e86-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.891955 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.891840    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c04bed-9587-4bd7-904c-cc0846d94e86-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.891955 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.891872    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c04bed-9587-4bd7-904c-cc0846d94e86-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.899944 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.899913    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:50.936799 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.935963    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84b94dcd67-zxvcc"]
Nov 01 05:51:50.941115 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:51:50.941084    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8797e16_282e_476e_8008_92456f432069.slice/crio-01e554585ccfaa5aec4d8aa75efc38b784fd93b555240981c5931257c14d5967 WatchSource:0}: Error finding container 01e554585ccfaa5aec4d8aa75efc38b784fd93b555240981c5931257c14d5967: Status 404 returned error can't find the container with id 01e554585ccfaa5aec4d8aa75efc38b784fd93b555240981c5931257c14d5967
Nov 01 05:51:50.992554 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.992510    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c04bed-9587-4bd7-904c-cc0846d94e86-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.992722 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.992588    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c04bed-9587-4bd7-904c-cc0846d94e86-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.992722 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.992622    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c04bed-9587-4bd7-904c-cc0846d94e86-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.992722 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.992667    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r7fw\" (UniqueName: \"kubernetes.io/projected/41c04bed-9587-4bd7-904c-cc0846d94e86-kube-api-access-5r7fw\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.992900 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.992727    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c04bed-9587-4bd7-904c-cc0846d94e86-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.992900 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.992758    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c04bed-9587-4bd7-904c-cc0846d94e86-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.992900 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.992785    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c04bed-9587-4bd7-904c-cc0846d94e86-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.992900 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.992833    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c04bed-9587-4bd7-904c-cc0846d94e86-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.992900 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.992866    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c04bed-9587-4bd7-904c-cc0846d94e86-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.993156 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.993002    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c04bed-9587-4bd7-904c-cc0846d94e86-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.993213 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.993165    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c04bed-9587-4bd7-904c-cc0846d94e86-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.993345 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.993303    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c04bed-9587-4bd7-904c-cc0846d94e86-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.993476 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.993427    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c04bed-9587-4bd7-904c-cc0846d94e86-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.993944 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.993917    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c04bed-9587-4bd7-904c-cc0846d94e86-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.996129 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.996107    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c04bed-9587-4bd7-904c-cc0846d94e86-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:50.996799 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:50.996776    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c04bed-9587-4bd7-904c-cc0846d94e86-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:51.051951 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:51.051874    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:51.057938 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:51.057908    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c04bed-9587-4bd7-904c-cc0846d94e86-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:51.058064 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:51.057973    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r7fw\" (UniqueName: \"kubernetes.io/projected/41c04bed-9587-4bd7-904c-cc0846d94e86-kube-api-access-5r7fw\") pod \"data-science-gateway-data-science-gateway-class-68d996c557tpltp\" (UID: \"41c04bed-9587-4bd7-904c-cc0846d94e86\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:51.073818 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:51.073791    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:51.307682 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:51.307599    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84b94dcd67-zxvcc" event={"ID":"e8797e16-282e-476e-8008-92456f432069","Type":"ContainerStarted","Data":"01e554585ccfaa5aec4d8aa75efc38b784fd93b555240981c5931257c14d5967"}
Nov 01 05:51:51.512225 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:51.512190    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"]
Nov 01 05:51:51.516041 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:51:51.516013    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c04bed_9587_4bd7_904c_cc0846d94e86.slice/crio-27ccb50580bd9685c471b2430ac58f910771b79277947afa4df16386efba08cf WatchSource:0}: Error finding container 27ccb50580bd9685c471b2430ac58f910771b79277947afa4df16386efba08cf: Status 404 returned error can't find the container with id 27ccb50580bd9685c471b2430ac58f910771b79277947afa4df16386efba08cf
Nov 01 05:51:51.518763 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:51.518724    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:51:51.518858 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:51.518815    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:51:51.518858 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:51.518845    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:51:52.231285 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:52.231251    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:52.314748 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:52.314708    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp" event={"ID":"41c04bed-9587-4bd7-904c-cc0846d94e86","Type":"ContainerStarted","Data":"863a07b86a213ed99d08f7bef03c4e620c9c763c33b2c8170077a5fa83837ee4"}
Nov 01 05:51:52.314919 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:52.314775    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp" event={"ID":"41c04bed-9587-4bd7-904c-cc0846d94e86","Type":"ContainerStarted","Data":"27ccb50580bd9685c471b2430ac58f910771b79277947afa4df16386efba08cf"}
Nov 01 05:51:52.371528 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:52.371483    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:52.441660 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:52.441593    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp" podStartSLOduration=2.441573767 podStartE2EDuration="2.441573767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:52.440265085 +0000 UTC m=+349.651088594" watchObservedRunningTime="2026-11-01 05:51:52.441573767 +0000 UTC m=+349.652397277"
Nov 01 05:51:52.713734 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:52.713702    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:53.044716 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:53.044572    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:53.074360 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:53.074316    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:53.079769 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:53.079726    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:53.185569 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:53.185532    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:53.273074 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:53.273044    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q"
Nov 01 05:51:53.320820 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:53.320737    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-zt4cd" event={"ID":"c3f69acb-521f-416f-9054-7fdf4ff8646a","Type":"ContainerStarted","Data":"6dfeadb7182fabea04754abbdb5b550829b836a0cd2790c37248d0ab1e519747"}
Nov 01 05:51:53.326688 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:53.322211    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:53.328667 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:53.328641    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557tpltp"
Nov 01 05:51:53.516921 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:53.516872    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-zt4cd" podStartSLOduration=2.094173645 podStartE2EDuration="6.516853511s" totalImagesPullingTime="4.422679866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:53.515543177 +0000 UTC m=+350.726366700" watchObservedRunningTime="2026-11-01 05:51:53.516853511 +0000 UTC m=+350.727677011"
Nov 01 05:51:54.622125 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:54.622065    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:54.770519 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:54.770440    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:55.331409 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:55.331368    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84b94dcd67-zxvcc" event={"ID":"e8797e16-282e-476e-8008-92456f432069","Type":"ContainerStarted","Data":"8b4bc8bb06eea2f7aa93bb2040e8fa867db27a01d12d64a983d132764b1cc004"}
Nov 01 05:51:55.453021 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:55.452970    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-84b94dcd67-zxvcc" podStartSLOduration=1.996126107 podStartE2EDuration="5.452956454s" totalImagesPullingTime="3.456830347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:51:55.451751352 +0000 UTC m=+352.662574861" watchObservedRunningTime="2026-11-01 05:51:55.452956454 +0000 UTC m=+352.663779960"
Nov 01 05:51:55.877910 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:51:55.877868    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:51:58.237296 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:58.237240    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:58.878751 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:58.878715    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:51:59.025180 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:51:59.025146    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:02.460947 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:02.460912    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:02.651692 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:02.651661    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:02.862467 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:02.862434    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:03.006311 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:03.006280    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:06.273781 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:06.273745    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:06.416070 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:06.416035    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:12.084916 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:12.084833    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:12.227359 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:12.227298    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:13.185569 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:13.185523    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:13.375017 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:13.374982    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:15.371301 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:15.371267    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:15.597682 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:15.597648    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:16.122196 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:16.122163    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-6fqw4"]
Nov 01 05:52:16.124372 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:16.124354    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6fqw4"
Nov 01 05:52:16.177776 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:16.177751    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 01 05:52:16.196591 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:16.196564    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-sgdcb\""
Nov 01 05:52:16.210661 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:16.210631    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:16.222722 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:16.222695    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p2kw\" (UniqueName: \"kubernetes.io/projected/8745c5be-e512-41f6-a186-8211cac046ee-kube-api-access-7p2kw\") pod \"postgres-7cdbf6dc44-6fqw4\" (UID: \"8745c5be-e512-41f6-a186-8211cac046ee\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6fqw4"
Nov 01 05:52:16.222809 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:16.222735    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8745c5be-e512-41f6-a186-8211cac046ee-data\") pod \"postgres-7cdbf6dc44-6fqw4\" (UID: \"8745c5be-e512-41f6-a186-8211cac046ee\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6fqw4"
Nov 01 05:52:16.228552 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:16.228532    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 05:52:16.228603 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:16.228551    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 05:52:16.229962 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:16.229944    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-6fqw4"]
Nov 01 05:52:16.324027 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:16.323993    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p2kw\" (UniqueName: \"kubernetes.io/projected/8745c5be-e512-41f6-a186-8211cac046ee-kube-api-access-7p2kw\") pod \"postgres-7cdbf6dc44-6fqw4\" (UID: \"8745c5be-e512-41f6-a186-8211cac046ee\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6fqw4"
Nov 01 05:52:16.324208 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:16.324041    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8745c5be-e512-41f6-a186-8211cac046ee-data\") pod \"postgres-7cdbf6dc44-6fqw4\" (UID: \"8745c5be-e512-41f6-a186-8211cac046ee\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6fqw4"
Nov 01 05:52:16.324435 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:16.324418    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8745c5be-e512-41f6-a186-8211cac046ee-data\") pod \"postgres-7cdbf6dc44-6fqw4\" (UID: \"8745c5be-e512-41f6-a186-8211cac046ee\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6fqw4"
Nov 01 05:52:16.385937 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:16.385876    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p2kw\" (UniqueName: \"kubernetes.io/projected/8745c5be-e512-41f6-a186-8211cac046ee-kube-api-access-7p2kw\") pod \"postgres-7cdbf6dc44-6fqw4\" (UID: \"8745c5be-e512-41f6-a186-8211cac046ee\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6fqw4"
Nov 01 05:52:16.435410 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:16.435384    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6fqw4"
Nov 01 05:52:16.803996 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:16.803918    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-6fqw4"]
Nov 01 05:52:16.807235 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:52:16.807205    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8745c5be_e512_41f6_a186_8211cac046ee.slice/crio-6ea7b4af953dec1e08e119f35dbeb9c5cb6c6b042c1450fc5e67429ae876923a WatchSource:0}: Error finding container 6ea7b4af953dec1e08e119f35dbeb9c5cb6c6b042c1450fc5e67429ae876923a: Status 404 returned error can't find the container with id 6ea7b4af953dec1e08e119f35dbeb9c5cb6c6b042c1450fc5e67429ae876923a
Nov 01 05:52:17.426514 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:17.426475    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6fqw4" event={"ID":"8745c5be-e512-41f6-a186-8211cac046ee","Type":"ContainerStarted","Data":"6ea7b4af953dec1e08e119f35dbeb9c5cb6c6b042c1450fc5e67429ae876923a"}
Nov 01 05:52:23.339563 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:23.339485    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:23.456127 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:23.456089    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6fqw4" event={"ID":"8745c5be-e512-41f6-a186-8211cac046ee","Type":"ContainerStarted","Data":"f7b8aa552d4d7499c213f2f8036595ffc55782f854754fe00cbc98239bef90ac"}
Nov 01 05:52:23.456317 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:23.456185    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6fqw4"
Nov 01 05:52:23.480421 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:23.480390    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:23.573979 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:23.573930    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6fqw4" podStartSLOduration=1.315140146 podStartE2EDuration="7.573917314s" totalImagesPullingTime="6.258777168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:52:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:52:23.573075423 +0000 UTC m=+380.783898933" watchObservedRunningTime="2026-11-01 05:52:23.573917314 +0000 UTC m=+380.784740822"
Nov 01 05:52:23.623973 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:23.623940    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:25.878877 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:52:25.878833    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:52:29.489053 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:29.489023    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6fqw4"
Nov 01 05:52:35.171076 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:35.171043    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:35.310281 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:35.310236    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:36.289168 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.289125    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:36.370669 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.370637    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f59fb4989-78468"]
Nov 01 05:52:36.372959 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.372943    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f59fb4989-78468"
Nov 01 05:52:36.425969 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.425942    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 05:52:36.438906 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.438878    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:36.476947 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.476921    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 05:52:36.477104 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.476956    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-htpqr\""
Nov 01 05:52:36.477104 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.476978    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 05:52:36.477270 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.477141    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 05:52:36.477270 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.477167    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 05:52:36.478411 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.478394    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f59fb4989-78468"]
Nov 01 05:52:36.496243 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.496224    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df672aac-f2c3-4a5e-9465-c3bd2be169c4-webhook-cert\") pod \"maas-controller-f59fb4989-78468\" (UID: \"df672aac-f2c3-4a5e-9465-c3bd2be169c4\") " pod="opendatahub/maas-controller-f59fb4989-78468"
Nov 01 05:52:36.496346 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.496251    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df672aac-f2c3-4a5e-9465-c3bd2be169c4-metrics-certs\") pod \"maas-controller-f59fb4989-78468\" (UID: \"df672aac-f2c3-4a5e-9465-c3bd2be169c4\") " pod="opendatahub/maas-controller-f59fb4989-78468"
Nov 01 05:52:36.496409 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.496391    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hzsc\" (UniqueName: \"kubernetes.io/projected/df672aac-f2c3-4a5e-9465-c3bd2be169c4-kube-api-access-5hzsc\") pod \"maas-controller-f59fb4989-78468\" (UID: \"df672aac-f2c3-4a5e-9465-c3bd2be169c4\") " pod="opendatahub/maas-controller-f59fb4989-78468"
Nov 01 05:52:36.584039 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.583958    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:36.597080 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.597055    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hzsc\" (UniqueName: \"kubernetes.io/projected/df672aac-f2c3-4a5e-9465-c3bd2be169c4-kube-api-access-5hzsc\") pod \"maas-controller-f59fb4989-78468\" (UID: \"df672aac-f2c3-4a5e-9465-c3bd2be169c4\") " pod="opendatahub/maas-controller-f59fb4989-78468"
Nov 01 05:52:36.597168 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.597103    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df672aac-f2c3-4a5e-9465-c3bd2be169c4-webhook-cert\") pod \"maas-controller-f59fb4989-78468\" (UID: \"df672aac-f2c3-4a5e-9465-c3bd2be169c4\") " pod="opendatahub/maas-controller-f59fb4989-78468"
Nov 01 05:52:36.597216 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.597189    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df672aac-f2c3-4a5e-9465-c3bd2be169c4-metrics-certs\") pod \"maas-controller-f59fb4989-78468\" (UID: \"df672aac-f2c3-4a5e-9465-c3bd2be169c4\") " pod="opendatahub/maas-controller-f59fb4989-78468"
Nov 01 05:52:36.599523 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.599501    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df672aac-f2c3-4a5e-9465-c3bd2be169c4-webhook-cert\") pod \"maas-controller-f59fb4989-78468\" (UID: \"df672aac-f2c3-4a5e-9465-c3bd2be169c4\") " pod="opendatahub/maas-controller-f59fb4989-78468"
Nov 01 05:52:36.599706 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.599686    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df672aac-f2c3-4a5e-9465-c3bd2be169c4-metrics-certs\") pod \"maas-controller-f59fb4989-78468\" (UID: \"df672aac-f2c3-4a5e-9465-c3bd2be169c4\") " pod="opendatahub/maas-controller-f59fb4989-78468"
Nov 01 05:52:36.659217 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.659194    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hzsc\" (UniqueName: \"kubernetes.io/projected/df672aac-f2c3-4a5e-9465-c3bd2be169c4-kube-api-access-5hzsc\") pod \"maas-controller-f59fb4989-78468\" (UID: \"df672aac-f2c3-4a5e-9465-c3bd2be169c4\") " pod="opendatahub/maas-controller-f59fb4989-78468"
Nov 01 05:52:36.684084 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.684060    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f59fb4989-78468"
Nov 01 05:52:36.731191 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.731140    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:36.939404 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.939374    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bf8b5db8-v4x7w"]
Nov 01 05:52:36.941975 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:36.941958    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bf8b5db8-v4x7w"
Nov 01 05:52:37.000380 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:37.000354    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkrbf\" (UniqueName: \"kubernetes.io/projected/f577a21b-1c78-4528-a1ea-a6a34c77260c-kube-api-access-nkrbf\") pod \"maas-controller-bf8b5db8-v4x7w\" (UID: \"f577a21b-1c78-4528-a1ea-a6a34c77260c\") " pod="opendatahub/maas-controller-bf8b5db8-v4x7w"
Nov 01 05:52:37.000532 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:37.000478    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f577a21b-1c78-4528-a1ea-a6a34c77260c-metrics-certs\") pod \"maas-controller-bf8b5db8-v4x7w\" (UID: \"f577a21b-1c78-4528-a1ea-a6a34c77260c\") " pod="opendatahub/maas-controller-bf8b5db8-v4x7w"
Nov 01 05:52:37.000580 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:37.000535    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f577a21b-1c78-4528-a1ea-a6a34c77260c-webhook-cert\") pod \"maas-controller-bf8b5db8-v4x7w\" (UID: \"f577a21b-1c78-4528-a1ea-a6a34c77260c\") " pod="opendatahub/maas-controller-bf8b5db8-v4x7w"
Nov 01 05:52:37.003210 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:37.003187    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bf8b5db8-v4x7w"]
Nov 01 05:52:37.054429 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:37.054401    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f59fb4989-78468"]
Nov 01 05:52:37.057151 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:52:37.057122    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf672aac_f2c3_4a5e_9465_c3bd2be169c4.slice/crio-69928e5054dcdef0762fe086335a02835a93c57ee9c7e38807d61b6d91331d07 WatchSource:0}: Error finding container 69928e5054dcdef0762fe086335a02835a93c57ee9c7e38807d61b6d91331d07: Status 404 returned error can't find the container with id 69928e5054dcdef0762fe086335a02835a93c57ee9c7e38807d61b6d91331d07
Nov 01 05:52:37.101522 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:37.101498    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkrbf\" (UniqueName: \"kubernetes.io/projected/f577a21b-1c78-4528-a1ea-a6a34c77260c-kube-api-access-nkrbf\") pod \"maas-controller-bf8b5db8-v4x7w\" (UID: \"f577a21b-1c78-4528-a1ea-a6a34c77260c\") " pod="opendatahub/maas-controller-bf8b5db8-v4x7w"
Nov 01 05:52:37.101649 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:37.101571    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f577a21b-1c78-4528-a1ea-a6a34c77260c-metrics-certs\") pod \"maas-controller-bf8b5db8-v4x7w\" (UID: \"f577a21b-1c78-4528-a1ea-a6a34c77260c\") " pod="opendatahub/maas-controller-bf8b5db8-v4x7w"
Nov 01 05:52:37.101649 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:37.101639    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f577a21b-1c78-4528-a1ea-a6a34c77260c-webhook-cert\") pod \"maas-controller-bf8b5db8-v4x7w\" (UID: \"f577a21b-1c78-4528-a1ea-a6a34c77260c\") " pod="opendatahub/maas-controller-bf8b5db8-v4x7w"
Nov 01 05:52:37.103906 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:37.103883    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f577a21b-1c78-4528-a1ea-a6a34c77260c-metrics-certs\") pod \"maas-controller-bf8b5db8-v4x7w\" (UID: \"f577a21b-1c78-4528-a1ea-a6a34c77260c\") " pod="opendatahub/maas-controller-bf8b5db8-v4x7w"
Nov 01 05:52:37.104010 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:37.103954    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f577a21b-1c78-4528-a1ea-a6a34c77260c-webhook-cert\") pod \"maas-controller-bf8b5db8-v4x7w\" (UID: \"f577a21b-1c78-4528-a1ea-a6a34c77260c\") " pod="opendatahub/maas-controller-bf8b5db8-v4x7w"
Nov 01 05:52:37.163885 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:37.163860    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkrbf\" (UniqueName: \"kubernetes.io/projected/f577a21b-1c78-4528-a1ea-a6a34c77260c-kube-api-access-nkrbf\") pod \"maas-controller-bf8b5db8-v4x7w\" (UID: \"f577a21b-1c78-4528-a1ea-a6a34c77260c\") " pod="opendatahub/maas-controller-bf8b5db8-v4x7w"
Nov 01 05:52:37.253012 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:37.252951    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bf8b5db8-v4x7w"
Nov 01 05:52:37.514285 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:37.514210    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f59fb4989-78468" event={"ID":"df672aac-f2c3-4a5e-9465-c3bd2be169c4","Type":"ContainerStarted","Data":"69928e5054dcdef0762fe086335a02835a93c57ee9c7e38807d61b6d91331d07"}
Nov 01 05:52:37.623355 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:37.623247    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bf8b5db8-v4x7w"]
Nov 01 05:52:37.626028 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:52:37.625999    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf577a21b_1c78_4528_a1ea_a6a34c77260c.slice/crio-e13efecaf5af4cbc98d10ae188d1c67d9c40a245aabed4476cba09d7ca09ce52 WatchSource:0}: Error finding container e13efecaf5af4cbc98d10ae188d1c67d9c40a245aabed4476cba09d7ca09ce52: Status 404 returned error can't find the container with id e13efecaf5af4cbc98d10ae188d1c67d9c40a245aabed4476cba09d7ca09ce52
Nov 01 05:52:38.519023 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:38.518990    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bf8b5db8-v4x7w" event={"ID":"f577a21b-1c78-4528-a1ea-a6a34c77260c","Type":"ContainerStarted","Data":"e13efecaf5af4cbc98d10ae188d1c67d9c40a245aabed4476cba09d7ca09ce52"}
Nov 01 05:52:44.971254 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:44.971216    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:45.118532 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:45.118486    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:45.561473 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:45.561437    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f59fb4989-78468" event={"ID":"df672aac-f2c3-4a5e-9465-c3bd2be169c4","Type":"ContainerStarted","Data":"eae73c109cbe5543e92f05d26c9a96d375342f937718e3182f3bd291d3f94e17"}
Nov 01 05:52:45.561654 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:45.561505    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f59fb4989-78468"
Nov 01 05:52:45.562906 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:45.562861    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bf8b5db8-v4x7w" event={"ID":"f577a21b-1c78-4528-a1ea-a6a34c77260c","Type":"ContainerStarted","Data":"8f2d8feec7246960a2114cb979591bc391b1173d70f43bde8a0dc8020f432910"}
Nov 01 05:52:45.563024 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:45.563005    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bf8b5db8-v4x7w"
Nov 01 05:52:45.703765 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:45.703715    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-f59fb4989-78468" podStartSLOduration=2.177545437 podStartE2EDuration="9.703700643s" totalImagesPullingTime="7.526155206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:52:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:52:45.701947631 +0000 UTC m=+402.912771153" watchObservedRunningTime="2026-11-01 05:52:45.703700643 +0000 UTC m=+402.914524189"
Nov 01 05:52:45.823094 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:45.823007    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-bf8b5db8-v4x7w" podStartSLOduration=2.824290144 podStartE2EDuration="9.82299276s" totalImagesPullingTime="6.998702616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:52:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:52:45.821550554 +0000 UTC m=+403.032374063" watchObservedRunningTime="2026-11-01 05:52:45.82299276 +0000 UTC m=+403.033816288"
Nov 01 05:52:48.065923 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:48.065883    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:48.269837 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:48.269802    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:52:55.879062 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:52:55.879024    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:52:56.568079 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:56.568048    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f59fb4989-78468"
Nov 01 05:52:56.568241 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:56.568145    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bf8b5db8-v4x7w"
Nov 01 05:52:56.878285 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:56.878246    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f59fb4989-78468"]
Nov 01 05:52:56.878525 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:56.878500    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-f59fb4989-78468" podUID="df672aac-f2c3-4a5e-9465-c3bd2be169c4" containerName="manager" containerID="cri-o://eae73c109cbe5543e92f05d26c9a96d375342f937718e3182f3bd291d3f94e17" gracePeriod=30
Nov 01 05:52:57.114943 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:57.114916    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f59fb4989-78468"
Nov 01 05:52:57.276357 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:57.276272    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df672aac-f2c3-4a5e-9465-c3bd2be169c4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df672aac-f2c3-4a5e-9465-c3bd2be169c4-metrics-certs\") pod \"df672aac-f2c3-4a5e-9465-c3bd2be169c4\" (UID: \"df672aac-f2c3-4a5e-9465-c3bd2be169c4\") "
Nov 01 05:52:57.276357 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:57.276356    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df672aac-f2c3-4a5e-9465-c3bd2be169c4-kube-api-access-5hzsc\" (UniqueName: \"kubernetes.io/projected/df672aac-f2c3-4a5e-9465-c3bd2be169c4-kube-api-access-5hzsc\") pod \"df672aac-f2c3-4a5e-9465-c3bd2be169c4\" (UID: \"df672aac-f2c3-4a5e-9465-c3bd2be169c4\") "
Nov 01 05:52:57.276583 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:57.276395    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df672aac-f2c3-4a5e-9465-c3bd2be169c4-webhook-cert\" (UniqueName: \"kubernetes.io/secret/df672aac-f2c3-4a5e-9465-c3bd2be169c4-webhook-cert\") pod \"df672aac-f2c3-4a5e-9465-c3bd2be169c4\" (UID: \"df672aac-f2c3-4a5e-9465-c3bd2be169c4\") "
Nov 01 05:52:57.278544 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:57.278512    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df672aac-f2c3-4a5e-9465-c3bd2be169c4-metrics-certs" pod "df672aac-f2c3-4a5e-9465-c3bd2be169c4" (UID: "df672aac-f2c3-4a5e-9465-c3bd2be169c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:52:57.278652 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:57.278565    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df672aac-f2c3-4a5e-9465-c3bd2be169c4-webhook-cert" pod "df672aac-f2c3-4a5e-9465-c3bd2be169c4" (UID: "df672aac-f2c3-4a5e-9465-c3bd2be169c4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:52:57.278849 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:57.278827    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df672aac-f2c3-4a5e-9465-c3bd2be169c4-kube-api-access-5hzsc" pod "df672aac-f2c3-4a5e-9465-c3bd2be169c4" (UID: "df672aac-f2c3-4a5e-9465-c3bd2be169c4"). InnerVolumeSpecName "kube-api-access-5hzsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:52:57.377303 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:57.377274    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hzsc\" (UniqueName: \"kubernetes.io/projected/df672aac-f2c3-4a5e-9465-c3bd2be169c4-kube-api-access-5hzsc\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:52:57.377303 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:57.377301    5619 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df672aac-f2c3-4a5e-9465-c3bd2be169c4-webhook-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:52:57.377499 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:57.377315    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df672aac-f2c3-4a5e-9465-c3bd2be169c4-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:52:57.612453 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:57.612419    5619 generic.go:356] "Generic (PLEG): container finished" podID="df672aac-f2c3-4a5e-9465-c3bd2be169c4" containerID="eae73c109cbe5543e92f05d26c9a96d375342f937718e3182f3bd291d3f94e17" exitCode=0
Nov 01 05:52:57.612591 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:57.612471    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f59fb4989-78468"
Nov 01 05:52:57.612591 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:57.612497    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f59fb4989-78468" event={"ID":"df672aac-f2c3-4a5e-9465-c3bd2be169c4","Type":"ContainerDied","Data":"eae73c109cbe5543e92f05d26c9a96d375342f937718e3182f3bd291d3f94e17"}
Nov 01 05:52:57.612591 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:57.612547    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f59fb4989-78468" event={"ID":"df672aac-f2c3-4a5e-9465-c3bd2be169c4","Type":"ContainerDied","Data":"69928e5054dcdef0762fe086335a02835a93c57ee9c7e38807d61b6d91331d07"}
Nov 01 05:52:57.612591 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:57.612569    5619 scope.go:122] "RemoveContainer" containerID="eae73c109cbe5543e92f05d26c9a96d375342f937718e3182f3bd291d3f94e17"
Nov 01 05:52:57.620819 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:57.620803    5619 scope.go:122] "RemoveContainer" containerID="eae73c109cbe5543e92f05d26c9a96d375342f937718e3182f3bd291d3f94e17"
Nov 01 05:52:57.621083 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:52:57.621061    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eae73c109cbe5543e92f05d26c9a96d375342f937718e3182f3bd291d3f94e17\": container with ID starting with eae73c109cbe5543e92f05d26c9a96d375342f937718e3182f3bd291d3f94e17 not found: ID does not exist" containerID="eae73c109cbe5543e92f05d26c9a96d375342f937718e3182f3bd291d3f94e17"
Nov 01 05:52:57.621158 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:57.621092    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae73c109cbe5543e92f05d26c9a96d375342f937718e3182f3bd291d3f94e17"} err="failed to get container status \"eae73c109cbe5543e92f05d26c9a96d375342f937718e3182f3bd291d3f94e17\": rpc error: code = NotFound desc = could not find container \"eae73c109cbe5543e92f05d26c9a96d375342f937718e3182f3bd291d3f94e17\": container with ID starting with eae73c109cbe5543e92f05d26c9a96d375342f937718e3182f3bd291d3f94e17 not found: ID does not exist"
Nov 01 05:52:57.779386 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:57.779361    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f59fb4989-78468"]
Nov 01 05:52:57.829896 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:57.829868    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f59fb4989-78468"]
Nov 01 05:52:58.960233 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:52:58.960202    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df672aac-f2c3-4a5e-9465-c3bd2be169c4" path="/var/lib/kubelet/pods/df672aac-f2c3-4a5e-9465-c3bd2be169c4/volumes"
Nov 01 05:53:09.090009 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:09.089969    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:09.282346 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:09.282277    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:09.439781 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:09.439751    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:09.587434 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:09.587401    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:09.807575 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:09.807507    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:09.959048 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:09.959012    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:10.019547 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.019516    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-zgwzb"]
Nov 01 05:53:10.019900 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.019887    5619 cpu_manager.go:414] "containerMap: removing container" podUID="df672aac-f2c3-4a5e-9465-c3bd2be169c4" containerName="manager"
Nov 01 05:53:10.019943 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.019901    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df672aac-f2c3-4a5e-9465-c3bd2be169c4" containerName="manager"
Nov 01 05:53:10.019997 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.019983    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="df672aac-f2c3-4a5e-9465-c3bd2be169c4" containerName="manager"
Nov 01 05:53:10.022600 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.022584    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-zgwzb"
Nov 01 05:53:10.108300 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.108269    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:10.126525 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.126502    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-frtnm\""
Nov 01 05:53:10.126640 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.126608    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 01 05:53:10.127828 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.127810    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-zgwzb"]
Nov 01 05:53:10.187992 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.187959    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9n6k\" (UniqueName: \"kubernetes.io/projected/02fae5ae-abd3-4895-a525-42ff3bae5ff4-kube-api-access-j9n6k\") pod \"kserve-controller-manager-776cc66d47-zgwzb\" (UID: \"02fae5ae-abd3-4895-a525-42ff3bae5ff4\") " pod="opendatahub/kserve-controller-manager-776cc66d47-zgwzb"
Nov 01 05:53:10.188097 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.188041    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02fae5ae-abd3-4895-a525-42ff3bae5ff4-cert\") pod \"kserve-controller-manager-776cc66d47-zgwzb\" (UID: \"02fae5ae-abd3-4895-a525-42ff3bae5ff4\") " pod="opendatahub/kserve-controller-manager-776cc66d47-zgwzb"
Nov 01 05:53:10.245037 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.245013    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-zvcc2"]
Nov 01 05:53:10.247420 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.247398    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-zvcc2"
Nov 01 05:53:10.258623 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.258591    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:10.289200 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.289175    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9n6k\" (UniqueName: \"kubernetes.io/projected/02fae5ae-abd3-4895-a525-42ff3bae5ff4-kube-api-access-j9n6k\") pod \"kserve-controller-manager-776cc66d47-zgwzb\" (UID: \"02fae5ae-abd3-4895-a525-42ff3bae5ff4\") " pod="opendatahub/kserve-controller-manager-776cc66d47-zgwzb"
Nov 01 05:53:10.289302 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.289220    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02fae5ae-abd3-4895-a525-42ff3bae5ff4-cert\") pod \"kserve-controller-manager-776cc66d47-zgwzb\" (UID: \"02fae5ae-abd3-4895-a525-42ff3bae5ff4\") " pod="opendatahub/kserve-controller-manager-776cc66d47-zgwzb"
Nov 01 05:53:10.291516 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.291497    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02fae5ae-abd3-4895-a525-42ff3bae5ff4-cert\") pod \"kserve-controller-manager-776cc66d47-zgwzb\" (UID: \"02fae5ae-abd3-4895-a525-42ff3bae5ff4\") " pod="opendatahub/kserve-controller-manager-776cc66d47-zgwzb"
Nov 01 05:53:10.300864 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.300841    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-28r8g\""
Nov 01 05:53:10.300946 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.300879    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 01 05:53:10.307336 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.307292    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-zvcc2"]
Nov 01 05:53:10.353153 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.353132    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9n6k\" (UniqueName: \"kubernetes.io/projected/02fae5ae-abd3-4895-a525-42ff3bae5ff4-kube-api-access-j9n6k\") pod \"kserve-controller-manager-776cc66d47-zgwzb\" (UID: \"02fae5ae-abd3-4895-a525-42ff3bae5ff4\") " pod="opendatahub/kserve-controller-manager-776cc66d47-zgwzb"
Nov 01 05:53:10.390365 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.390304    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d763214-1e93-47b0-bfb4-e3321fc6d742-tls-certs\") pod \"model-serving-api-75bd89fdfc-zvcc2\" (UID: \"9d763214-1e93-47b0-bfb4-e3321fc6d742\") " pod="opendatahub/model-serving-api-75bd89fdfc-zvcc2"
Nov 01 05:53:10.390365 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.390346    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkgdq\" (UniqueName: \"kubernetes.io/projected/9d763214-1e93-47b0-bfb4-e3321fc6d742-kube-api-access-pkgdq\") pod \"model-serving-api-75bd89fdfc-zvcc2\" (UID: \"9d763214-1e93-47b0-bfb4-e3321fc6d742\") " pod="opendatahub/model-serving-api-75bd89fdfc-zvcc2"
Nov 01 05:53:10.491875 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.491846    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d763214-1e93-47b0-bfb4-e3321fc6d742-tls-certs\") pod \"model-serving-api-75bd89fdfc-zvcc2\" (UID: \"9d763214-1e93-47b0-bfb4-e3321fc6d742\") " pod="opendatahub/model-serving-api-75bd89fdfc-zvcc2"
Nov 01 05:53:10.491875 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.491877    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkgdq\" (UniqueName: \"kubernetes.io/projected/9d763214-1e93-47b0-bfb4-e3321fc6d742-kube-api-access-pkgdq\") pod \"model-serving-api-75bd89fdfc-zvcc2\" (UID: \"9d763214-1e93-47b0-bfb4-e3321fc6d742\") " pod="opendatahub/model-serving-api-75bd89fdfc-zvcc2"
Nov 01 05:53:10.494241 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.494224    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d763214-1e93-47b0-bfb4-e3321fc6d742-tls-certs\") pod \"model-serving-api-75bd89fdfc-zvcc2\" (UID: \"9d763214-1e93-47b0-bfb4-e3321fc6d742\") " pod="opendatahub/model-serving-api-75bd89fdfc-zvcc2"
Nov 01 05:53:10.553726 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.553701    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkgdq\" (UniqueName: \"kubernetes.io/projected/9d763214-1e93-47b0-bfb4-e3321fc6d742-kube-api-access-pkgdq\") pod \"model-serving-api-75bd89fdfc-zvcc2\" (UID: \"9d763214-1e93-47b0-bfb4-e3321fc6d742\") " pod="opendatahub/model-serving-api-75bd89fdfc-zvcc2"
Nov 01 05:53:10.558469 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.558439    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-zvcc2"
Nov 01 05:53:10.633697 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.633666    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-zgwzb"
Nov 01 05:53:10.926274 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:10.926244    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-zvcc2"]
Nov 01 05:53:10.928805 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:53:10.928778    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d763214_1e93_47b0_bfb4_e3321fc6d742.slice/crio-c50664897fb3494c984a299f5bb59197abd072e50e740da7716830797cf37cca WatchSource:0}: Error finding container c50664897fb3494c984a299f5bb59197abd072e50e740da7716830797cf37cca: Status 404 returned error can't find the container with id c50664897fb3494c984a299f5bb59197abd072e50e740da7716830797cf37cca
Nov 01 05:53:11.047155 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:11.047033    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-zgwzb"]
Nov 01 05:53:11.049693 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:53:11.049668    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02fae5ae_abd3_4895_a525_42ff3bae5ff4.slice/crio-a0d3533e66c8b5a5cf1570a8d3554df4f2c50a0e1a30475ecb42870bd6b7af3f WatchSource:0}: Error finding container a0d3533e66c8b5a5cf1570a8d3554df4f2c50a0e1a30475ecb42870bd6b7af3f: Status 404 returned error can't find the container with id a0d3533e66c8b5a5cf1570a8d3554df4f2c50a0e1a30475ecb42870bd6b7af3f
Nov 01 05:53:11.385235 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:11.385205    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:11.674742 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:11.674651    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-zgwzb" event={"ID":"02fae5ae-abd3-4895-a525-42ff3bae5ff4","Type":"ContainerStarted","Data":"a0d3533e66c8b5a5cf1570a8d3554df4f2c50a0e1a30475ecb42870bd6b7af3f"}
Nov 01 05:53:11.675728 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:11.675696    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-zvcc2" event={"ID":"9d763214-1e93-47b0-bfb4-e3321fc6d742","Type":"ContainerStarted","Data":"c50664897fb3494c984a299f5bb59197abd072e50e740da7716830797cf37cca"}
Nov 01 05:53:13.522533 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:13.522498    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:14.692679 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:14.692646    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-zvcc2" event={"ID":"9d763214-1e93-47b0-bfb4-e3321fc6d742","Type":"ContainerStarted","Data":"8658d187b61c656fd8b13bc6078b9eea5551ed73b10913fbb4f4cd445621932c"}
Nov 01 05:53:14.693041 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:14.692752    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-zvcc2"
Nov 01 05:53:14.812031 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:14.811968    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-zvcc2" podStartSLOduration=1.6271389649999999 podStartE2EDuration="4.811953267s" totalImagesPullingTime="3.184814302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:53:14.810590177 +0000 UTC m=+432.021413687" watchObservedRunningTime="2026-11-01 05:53:14.811953267 +0000 UTC m=+432.022776776"
Nov 01 05:53:16.699693 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:16.699631    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:16.707142 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:16.707105    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-zgwzb" event={"ID":"02fae5ae-abd3-4895-a525-42ff3bae5ff4","Type":"ContainerStarted","Data":"9ade34dde336076fa16c21701f35bef0924e1903a802d8deb0ff5396de4778f3"}
Nov 01 05:53:16.707371 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:16.707353    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-zgwzb"
Nov 01 05:53:16.825482 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:16.825378    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-zgwzb" podStartSLOduration=2.4089451410000002 podStartE2EDuration="7.825363639s" totalImagesPullingTime="5.416418498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:53:16.824755563 +0000 UTC m=+434.035579066" watchObservedRunningTime="2026-11-01 05:53:16.825363639 +0000 UTC m=+434.036187147"
Nov 01 05:53:16.843346 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:16.843302    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:16.988108 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:16.988080    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:17.130848 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:17.130808    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:17.271200 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:17.271143    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:17.474188 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:17.474107    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:17.675752 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:17.675713    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:17.818119 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:17.818037    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:17.961620 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:17.961584    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:18.329336 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:18.329290    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:18.471705 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:18.471674    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:20.402934 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:20.402896    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:20.610794 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:20.610761    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:20.752256 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:20.752170    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:20.893272 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:20.893241    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:21.462544 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:21.462512    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"]
Nov 01 05:53:21.464751 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:21.464734    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"
Nov 01 05:53:21.490100 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:21.490075    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5vhj\" (UniqueName: \"kubernetes.io/projected/1b66a44d-1862-4205-b966-133f40f97347-kube-api-access-r5vhj\") pod \"maas-api-8449957c6b-4fzbw\" (UID: \"1b66a44d-1862-4205-b966-133f40f97347\") " pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"
Nov 01 05:53:21.490212 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:21.490122    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b66a44d-1862-4205-b966-133f40f97347-maas-api-tls\") pod \"maas-api-8449957c6b-4fzbw\" (UID: \"1b66a44d-1862-4205-b966-133f40f97347\") " pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"
Nov 01 05:53:21.490252 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:21.490237    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b66a44d-1862-4205-b966-133f40f97347-metrics-certs\") pod \"maas-api-8449957c6b-4fzbw\" (UID: \"1b66a44d-1862-4205-b966-133f40f97347\") " pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"
Nov 01 05:53:21.490295 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:21.490272    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b66a44d-1862-4205-b966-133f40f97347-openshift-service-ca\") pod \"maas-api-8449957c6b-4fzbw\" (UID: \"1b66a44d-1862-4205-b966-133f40f97347\") " pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"
Nov 01 05:53:21.568960 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:21.568932    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 05:53:21.569123 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:21.569049    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 01 05:53:21.569123 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:21.569073    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rqph5\""
Nov 01 05:53:21.570210 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:21.570188    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"]
Nov 01 05:53:21.591056 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:21.591016    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b66a44d-1862-4205-b966-133f40f97347-openshift-service-ca\") pod \"maas-api-8449957c6b-4fzbw\" (UID: \"1b66a44d-1862-4205-b966-133f40f97347\") " pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"
Nov 01 05:53:21.591207 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:21.591093    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5vhj\" (UniqueName: \"kubernetes.io/projected/1b66a44d-1862-4205-b966-133f40f97347-kube-api-access-r5vhj\") pod \"maas-api-8449957c6b-4fzbw\" (UID: \"1b66a44d-1862-4205-b966-133f40f97347\") " pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"
Nov 01 05:53:21.591207 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:21.591166    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b66a44d-1862-4205-b966-133f40f97347-maas-api-tls\") pod \"maas-api-8449957c6b-4fzbw\" (UID: \"1b66a44d-1862-4205-b966-133f40f97347\") " pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"
Nov 01 05:53:21.591357 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:21.591235    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b66a44d-1862-4205-b966-133f40f97347-metrics-certs\") pod \"maas-api-8449957c6b-4fzbw\" (UID: \"1b66a44d-1862-4205-b966-133f40f97347\") " pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"
Nov 01 05:53:21.591804 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:21.591780    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b66a44d-1862-4205-b966-133f40f97347-openshift-service-ca\") pod \"maas-api-8449957c6b-4fzbw\" (UID: \"1b66a44d-1862-4205-b966-133f40f97347\") " pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"
Nov 01 05:53:21.593634 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:21.593612    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b66a44d-1862-4205-b966-133f40f97347-maas-api-tls\") pod \"maas-api-8449957c6b-4fzbw\" (UID: \"1b66a44d-1862-4205-b966-133f40f97347\") " pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"
Nov 01 05:53:21.593758 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:21.593739    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b66a44d-1862-4205-b966-133f40f97347-metrics-certs\") pod \"maas-api-8449957c6b-4fzbw\" (UID: \"1b66a44d-1862-4205-b966-133f40f97347\") " pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"
Nov 01 05:53:21.652837 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:21.652810    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5vhj\" (UniqueName: \"kubernetes.io/projected/1b66a44d-1862-4205-b966-133f40f97347-kube-api-access-r5vhj\") pod \"maas-api-8449957c6b-4fzbw\" (UID: \"1b66a44d-1862-4205-b966-133f40f97347\") " pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"
Nov 01 05:53:21.775997 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:21.775931    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"
Nov 01 05:53:22.145426 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:22.145257    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"]
Nov 01 05:53:22.148029 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:53:22.148002    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b66a44d_1862_4205_b966_133f40f97347.slice/crio-df174b0ac751d9ffb6b2afa49bfaeac0574c2578bb173b38dfe7f861c0809150 WatchSource:0}: Error finding container df174b0ac751d9ffb6b2afa49bfaeac0574c2578bb173b38dfe7f861c0809150: Status 404 returned error can't find the container with id df174b0ac751d9ffb6b2afa49bfaeac0574c2578bb173b38dfe7f861c0809150
Nov 01 05:53:22.732067 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:22.732031    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw" event={"ID":"1b66a44d-1862-4205-b966-133f40f97347","Type":"ContainerStarted","Data":"df174b0ac751d9ffb6b2afa49bfaeac0574c2578bb173b38dfe7f861c0809150"}
Nov 01 05:53:25.700796 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:25.700767    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-zvcc2"
Nov 01 05:53:25.886424 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:53:25.886384    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:53:28.759952 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:28.759920    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw" event={"ID":"1b66a44d-1862-4205-b966-133f40f97347","Type":"ContainerStarted","Data":"c1572a9dfbca2b9e536fadfa7cae9c755503a42159b9f219a72a188dfe8c6597"}
Nov 01 05:53:28.760338 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:28.760043    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"
Nov 01 05:53:28.879056 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:28.879014    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw" podStartSLOduration=2.182549511 podStartE2EDuration="7.879000454s" totalImagesPullingTime="5.696450943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:53:28.87848466 +0000 UTC m=+446.089308190" watchObservedRunningTime="2026-11-01 05:53:28.879000454 +0000 UTC m=+446.089823963"
Nov 01 05:53:34.769342 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:34.769292    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"
Nov 01 05:53:35.287535 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:35.287504    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:35.504092 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:35.504050    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:35.762202 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:35.762167    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"]
Nov 01 05:53:35.765807 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:35.765786    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"
Nov 01 05:53:35.825439 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:35.825414    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"]
Nov 01 05:53:35.914727 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:35.914699    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqvmt\" (UniqueName: \"kubernetes.io/projected/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-kube-api-access-wqvmt\") pod \"maas-api-bdb66f487-jxkws\" (UID: \"c0ecdd10-e2e4-4956-a985-741f86e7f3a6\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"
Nov 01 05:53:35.914883 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:35.914742    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-openshift-service-ca\") pod \"maas-api-bdb66f487-jxkws\" (UID: \"c0ecdd10-e2e4-4956-a985-741f86e7f3a6\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"
Nov 01 05:53:35.914883 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:35.914854    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-metrics-certs\") pod \"maas-api-bdb66f487-jxkws\" (UID: \"c0ecdd10-e2e4-4956-a985-741f86e7f3a6\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"
Nov 01 05:53:35.914960 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:35.914910    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-maas-api-tls\") pod \"maas-api-bdb66f487-jxkws\" (UID: \"c0ecdd10-e2e4-4956-a985-741f86e7f3a6\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"
Nov 01 05:53:36.016090 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:36.016008    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqvmt\" (UniqueName: \"kubernetes.io/projected/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-kube-api-access-wqvmt\") pod \"maas-api-bdb66f487-jxkws\" (UID: \"c0ecdd10-e2e4-4956-a985-741f86e7f3a6\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"
Nov 01 05:53:36.016090 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:36.016073    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-openshift-service-ca\") pod \"maas-api-bdb66f487-jxkws\" (UID: \"c0ecdd10-e2e4-4956-a985-741f86e7f3a6\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"
Nov 01 05:53:36.016313 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:36.016163    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-metrics-certs\") pod \"maas-api-bdb66f487-jxkws\" (UID: \"c0ecdd10-e2e4-4956-a985-741f86e7f3a6\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"
Nov 01 05:53:36.016313 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:36.016259    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-maas-api-tls\") pod \"maas-api-bdb66f487-jxkws\" (UID: \"c0ecdd10-e2e4-4956-a985-741f86e7f3a6\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"
Nov 01 05:53:36.016905 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:36.016877    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-openshift-service-ca\") pod \"maas-api-bdb66f487-jxkws\" (UID: \"c0ecdd10-e2e4-4956-a985-741f86e7f3a6\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"
Nov 01 05:53:36.018862 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:36.018834    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-metrics-certs\") pod \"maas-api-bdb66f487-jxkws\" (UID: \"c0ecdd10-e2e4-4956-a985-741f86e7f3a6\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"
Nov 01 05:53:36.018961 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:36.018844    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-maas-api-tls\") pod \"maas-api-bdb66f487-jxkws\" (UID: \"c0ecdd10-e2e4-4956-a985-741f86e7f3a6\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"
Nov 01 05:53:36.078254 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:36.078228    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqvmt\" (UniqueName: \"kubernetes.io/projected/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-kube-api-access-wqvmt\") pod \"maas-api-bdb66f487-jxkws\" (UID: \"c0ecdd10-e2e4-4956-a985-741f86e7f3a6\") " pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"
Nov 01 05:53:36.378726 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:36.378700    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"
Nov 01 05:53:36.754935 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:36.754859    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"]
Nov 01 05:53:36.758023 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:53:36.757995    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ecdd10_e2e4_4956_a985_741f86e7f3a6.slice/crio-76bafb82de183bb8814a3d0f0d9378ca035fddacac71398340916423bbb1c959 WatchSource:0}: Error finding container 76bafb82de183bb8814a3d0f0d9378ca035fddacac71398340916423bbb1c959: Status 404 returned error can't find the container with id 76bafb82de183bb8814a3d0f0d9378ca035fddacac71398340916423bbb1c959
Nov 01 05:53:36.793749 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:36.793714    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws" event={"ID":"c0ecdd10-e2e4-4956-a985-741f86e7f3a6","Type":"ContainerStarted","Data":"76bafb82de183bb8814a3d0f0d9378ca035fddacac71398340916423bbb1c959"}
Nov 01 05:53:37.799315 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:37.799278    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws" event={"ID":"c0ecdd10-e2e4-4956-a985-741f86e7f3a6","Type":"ContainerStarted","Data":"80bd178d909dc05fd46b27959aba6e99a6f2e4bc6f863511f6095b271722fc98"}
Nov 01 05:53:37.799706 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:37.799366    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"
Nov 01 05:53:37.920369 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:37.920299    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws" podStartSLOduration=2.089420523 podStartE2EDuration="2.92028591s" totalImagesPullingTime="830.865387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:53:37.918996191 +0000 UTC m=+455.129819701" watchObservedRunningTime="2026-11-01 05:53:37.92028591 +0000 UTC m=+455.131109418"
Nov 01 05:53:43.808935 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:43.808858    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"
Nov 01 05:53:43.980580 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:43.980549    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"]
Nov 01 05:53:43.980780 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:43.980755    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw" podUID="1b66a44d-1862-4205-b966-133f40f97347" containerName="maas-api" containerID="cri-o://c1572a9dfbca2b9e536fadfa7cae9c755503a42159b9f219a72a188dfe8c6597" gracePeriod=30
Nov 01 05:53:44.225089 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:44.225052    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"
Nov 01 05:53:44.387700 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:44.387631    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b66a44d-1862-4205-b966-133f40f97347-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b66a44d-1862-4205-b966-133f40f97347-metrics-certs\") pod \"1b66a44d-1862-4205-b966-133f40f97347\" (UID: \"1b66a44d-1862-4205-b966-133f40f97347\") "
Nov 01 05:53:44.387700 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:44.387691    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b66a44d-1862-4205-b966-133f40f97347-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b66a44d-1862-4205-b966-133f40f97347-maas-api-tls\") pod \"1b66a44d-1862-4205-b966-133f40f97347\" (UID: \"1b66a44d-1862-4205-b966-133f40f97347\") "
Nov 01 05:53:44.387885 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:44.387745    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b66a44d-1862-4205-b966-133f40f97347-kube-api-access-r5vhj\" (UniqueName: \"kubernetes.io/projected/1b66a44d-1862-4205-b966-133f40f97347-kube-api-access-r5vhj\") pod \"1b66a44d-1862-4205-b966-133f40f97347\" (UID: \"1b66a44d-1862-4205-b966-133f40f97347\") "
Nov 01 05:53:44.387885 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:44.387767    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b66a44d-1862-4205-b966-133f40f97347-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b66a44d-1862-4205-b966-133f40f97347-openshift-service-ca\") pod \"1b66a44d-1862-4205-b966-133f40f97347\" (UID: \"1b66a44d-1862-4205-b966-133f40f97347\") "
Nov 01 05:53:44.388197 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:44.388163    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b66a44d-1862-4205-b966-133f40f97347-openshift-service-ca" pod "1b66a44d-1862-4205-b966-133f40f97347" (UID: "1b66a44d-1862-4205-b966-133f40f97347"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 05:53:44.389875 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:44.389842    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b66a44d-1862-4205-b966-133f40f97347-kube-api-access-r5vhj" pod "1b66a44d-1862-4205-b966-133f40f97347" (UID: "1b66a44d-1862-4205-b966-133f40f97347"). InnerVolumeSpecName "kube-api-access-r5vhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 05:53:44.389971 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:44.389855    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b66a44d-1862-4205-b966-133f40f97347-maas-api-tls" pod "1b66a44d-1862-4205-b966-133f40f97347" (UID: "1b66a44d-1862-4205-b966-133f40f97347"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:53:44.389971 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:44.389873    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b66a44d-1862-4205-b966-133f40f97347-metrics-certs" pod "1b66a44d-1862-4205-b966-133f40f97347" (UID: "1b66a44d-1862-4205-b966-133f40f97347"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 05:53:44.489432 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:44.489406    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b66a44d-1862-4205-b966-133f40f97347-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:44.489432 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:44.489432    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b66a44d-1862-4205-b966-133f40f97347-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:44.489618 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:44.489447    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5vhj\" (UniqueName: \"kubernetes.io/projected/1b66a44d-1862-4205-b966-133f40f97347-kube-api-access-r5vhj\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:44.489618 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:44.489462    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b66a44d-1862-4205-b966-133f40f97347-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 05:53:44.828110 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:44.828030    5619 generic.go:356] "Generic (PLEG): container finished" podID="1b66a44d-1862-4205-b966-133f40f97347" containerID="c1572a9dfbca2b9e536fadfa7cae9c755503a42159b9f219a72a188dfe8c6597" exitCode=0
Nov 01 05:53:44.828110 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:44.828077    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"
Nov 01 05:53:44.828545 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:44.828111    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw" event={"ID":"1b66a44d-1862-4205-b966-133f40f97347","Type":"ContainerDied","Data":"c1572a9dfbca2b9e536fadfa7cae9c755503a42159b9f219a72a188dfe8c6597"}
Nov 01 05:53:44.828545 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:44.828165    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw" event={"ID":"1b66a44d-1862-4205-b966-133f40f97347","Type":"ContainerDied","Data":"df174b0ac751d9ffb6b2afa49bfaeac0574c2578bb173b38dfe7f861c0809150"}
Nov 01 05:53:44.828545 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:44.828187    5619 scope.go:122] "RemoveContainer" containerID="c1572a9dfbca2b9e536fadfa7cae9c755503a42159b9f219a72a188dfe8c6597"
Nov 01 05:53:44.837146 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:44.837129    5619 scope.go:122] "RemoveContainer" containerID="c1572a9dfbca2b9e536fadfa7cae9c755503a42159b9f219a72a188dfe8c6597"
Nov 01 05:53:44.837422 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:53:44.837404    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1572a9dfbca2b9e536fadfa7cae9c755503a42159b9f219a72a188dfe8c6597\": container with ID starting with c1572a9dfbca2b9e536fadfa7cae9c755503a42159b9f219a72a188dfe8c6597 not found: ID does not exist" containerID="c1572a9dfbca2b9e536fadfa7cae9c755503a42159b9f219a72a188dfe8c6597"
Nov 01 05:53:44.837502 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:44.837444    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1572a9dfbca2b9e536fadfa7cae9c755503a42159b9f219a72a188dfe8c6597"} err="failed to get container status \"c1572a9dfbca2b9e536fadfa7cae9c755503a42159b9f219a72a188dfe8c6597\": rpc error: code = NotFound desc = could not find container \"c1572a9dfbca2b9e536fadfa7cae9c755503a42159b9f219a72a188dfe8c6597\": container with ID starting with c1572a9dfbca2b9e536fadfa7cae9c755503a42159b9f219a72a188dfe8c6597 not found: ID does not exist"
Nov 01 05:53:44.909293 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:44.909260    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:44.995093 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:44.995061    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"]
Nov 01 05:53:45.046001 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:45.045968    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-8449957c6b-4fzbw"]
Nov 01 05:53:46.959875 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:46.959841    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b66a44d-1862-4205-b966-133f40f97347" path="/var/lib/kubelet/pods/1b66a44d-1862-4205-b966-133f40f97347/volumes"
Nov 01 05:53:47.449634 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:47.449595    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:53:47.716414 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:53:47.716339    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-zgwzb"
Nov 01 05:53:55.878674 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:53:55.878635    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:54:13.886525 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:54:13.886479    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:25.879775 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:54:25.879739    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:54:54.322991 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:54:54.322958    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:54.652860 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:54:54.652828    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:54.867682 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:54:54.867650    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:54:55.877879 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:54:55.877848    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:55:01.155353 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:01.155309    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:01.345013 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:01.344973    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:01.717615 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:01.717580    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-8444n"]
Nov 01 05:55:01.717941 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:01.717928    5619 cpu_manager.go:414] "containerMap: removing container" podUID="1b66a44d-1862-4205-b966-133f40f97347" containerName="maas-api"
Nov 01 05:55:01.717983 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:01.717941    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b66a44d-1862-4205-b966-133f40f97347" containerName="maas-api"
Nov 01 05:55:01.718018 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:01.718010    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b66a44d-1862-4205-b966-133f40f97347" containerName="maas-api"
Nov 01 05:55:01.720946 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:01.720930    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-8444n"
Nov 01 05:55:01.736943 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:01.736910    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khtmv\" (UniqueName: \"kubernetes.io/projected/1f121a97-b0e6-422d-ac1b-f3a235b333e9-kube-api-access-khtmv\") pod \"payload-processing-55fbbbf9dc-8444n\" (UID: \"1f121a97-b0e6-422d-ac1b-f3a235b333e9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-8444n"
Nov 01 05:55:01.737075 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:01.736965    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f121a97-b0e6-422d-ac1b-f3a235b333e9-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-8444n\" (UID: \"1f121a97-b0e6-422d-ac1b-f3a235b333e9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-8444n"
Nov 01 05:55:01.824989 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:01.824958    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 01 05:55:01.824989 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:01.824958    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-7d8fb\""
Nov 01 05:55:01.826253 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:01.826233    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-8444n"]
Nov 01 05:55:01.838225 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:01.838194    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khtmv\" (UniqueName: \"kubernetes.io/projected/1f121a97-b0e6-422d-ac1b-f3a235b333e9-kube-api-access-khtmv\") pod \"payload-processing-55fbbbf9dc-8444n\" (UID: \"1f121a97-b0e6-422d-ac1b-f3a235b333e9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-8444n"
Nov 01 05:55:01.838383 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:01.838233    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f121a97-b0e6-422d-ac1b-f3a235b333e9-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-8444n\" (UID: \"1f121a97-b0e6-422d-ac1b-f3a235b333e9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-8444n"
Nov 01 05:55:01.838997 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:01.838978    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f121a97-b0e6-422d-ac1b-f3a235b333e9-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-8444n\" (UID: \"1f121a97-b0e6-422d-ac1b-f3a235b333e9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-8444n"
Nov 01 05:55:01.900459 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:01.900421    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khtmv\" (UniqueName: \"kubernetes.io/projected/1f121a97-b0e6-422d-ac1b-f3a235b333e9-kube-api-access-khtmv\") pod \"payload-processing-55fbbbf9dc-8444n\" (UID: \"1f121a97-b0e6-422d-ac1b-f3a235b333e9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-8444n"
Nov 01 05:55:02.033694 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:02.033613    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-8444n"
Nov 01 05:55:02.405881 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:02.405686    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-8444n"]
Nov 01 05:55:02.408823 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:55:02.408793    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f121a97_b0e6_422d_ac1b_f3a235b333e9.slice/crio-db8dbc7ac66bd5cb54427ef23d595fc500a4bc38e8e00f59346e7ffbc0447023 WatchSource:0}: Error finding container db8dbc7ac66bd5cb54427ef23d595fc500a4bc38e8e00f59346e7ffbc0447023: Status 404 returned error can't find the container with id db8dbc7ac66bd5cb54427ef23d595fc500a4bc38e8e00f59346e7ffbc0447023
Nov 01 05:55:03.169654 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:03.169621    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-8444n" event={"ID":"1f121a97-b0e6-422d-ac1b-f3a235b333e9","Type":"ContainerStarted","Data":"db8dbc7ac66bd5cb54427ef23d595fc500a4bc38e8e00f59346e7ffbc0447023"}
Nov 01 05:55:05.179738 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:05.179703    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-8444n" event={"ID":"1f121a97-b0e6-422d-ac1b-f3a235b333e9","Type":"ContainerStarted","Data":"a6e9c75b0971bcda5263381ee0650dd72ed140271d390169c6910811e2f30aea"}
Nov 01 05:55:05.180129 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:05.179852    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-8444n"
Nov 01 05:55:05.298647 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:05.298549    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-8444n" podStartSLOduration=1.7642446029999999 podStartE2EDuration="4.298532036s" totalImagesPullingTime="2.534287433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:55:05.298449496 +0000 UTC m=+542.509273016" watchObservedRunningTime="2026-11-01 05:55:05.298532036 +0000 UTC m=+542.509355545"
Nov 01 05:55:12.806720 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:12.806635    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:14.223443 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:14.223400    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:15.915400 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:15.915365    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:16.188627 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:16.188553    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-8444n"
Nov 01 05:55:16.227202 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:16.227173    5619 generic.go:356] "Generic (PLEG): container finished" podID="f577a21b-1c78-4528-a1ea-a6a34c77260c" containerID="8f2d8feec7246960a2114cb979591bc391b1173d70f43bde8a0dc8020f432910" exitCode=1
Nov 01 05:55:16.227345 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:16.227225    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bf8b5db8-v4x7w" event={"ID":"f577a21b-1c78-4528-a1ea-a6a34c77260c","Type":"ContainerDied","Data":"8f2d8feec7246960a2114cb979591bc391b1173d70f43bde8a0dc8020f432910"}
Nov 01 05:55:16.227531 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:16.227517    5619 scope.go:122] "RemoveContainer" containerID="8f2d8feec7246960a2114cb979591bc391b1173d70f43bde8a0dc8020f432910"
Nov 01 05:55:16.566671 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:16.566589    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bf8b5db8-v4x7w"
Nov 01 05:55:17.233739 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:17.233704    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bf8b5db8-v4x7w" event={"ID":"f577a21b-1c78-4528-a1ea-a6a34c77260c","Type":"ContainerStarted","Data":"f175553e0a74bbc6912196d734d95f9b1111360cee11f348c0548766ad9d1732"}
Nov 01 05:55:17.234103 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:17.233835    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bf8b5db8-v4x7w"
Nov 01 05:55:25.298117 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:25.298080    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:25.878345 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:55:25.878290    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:55:28.240311 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:28.240283    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bf8b5db8-v4x7w"
Nov 01 05:55:35.750237 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:35.750199    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:37.005841 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:37.005805    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:37.196870 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:37.196834    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:55.878350 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:55:55.878300    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:55:55.968666 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:55.968632    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"]
Nov 01 05:55:55.972242 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:55.972226    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:55:56.016186 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.016143    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1951f4ec-2955-4b93-a40e-34de8400b217-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f\" (UID: \"1951f4ec-2955-4b93-a40e-34de8400b217\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:55:56.016386 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.016301    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1951f4ec-2955-4b93-a40e-34de8400b217-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f\" (UID: \"1951f4ec-2955-4b93-a40e-34de8400b217\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:55:56.016386 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.016371    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1951f4ec-2955-4b93-a40e-34de8400b217-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f\" (UID: \"1951f4ec-2955-4b93-a40e-34de8400b217\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:55:56.016523 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.016412    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1951f4ec-2955-4b93-a40e-34de8400b217-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f\" (UID: \"1951f4ec-2955-4b93-a40e-34de8400b217\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:55:56.016587 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.016515    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1951f4ec-2955-4b93-a40e-34de8400b217-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f\" (UID: \"1951f4ec-2955-4b93-a40e-34de8400b217\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:55:56.016766 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.016661    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fnxz\" (UniqueName: \"kubernetes.io/projected/1951f4ec-2955-4b93-a40e-34de8400b217-kube-api-access-7fnxz\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f\" (UID: \"1951f4ec-2955-4b93-a40e-34de8400b217\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:55:56.046089 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.046064    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 01 05:55:56.060674 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.060641    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:55:56.076224 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.076198    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 01 05:55:56.076474 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.076364    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 01 05:55:56.076474 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.076371    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4k6qc\""
Nov 01 05:55:56.077776 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.077754    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"]
Nov 01 05:55:56.117737 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.117708    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fnxz\" (UniqueName: \"kubernetes.io/projected/1951f4ec-2955-4b93-a40e-34de8400b217-kube-api-access-7fnxz\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f\" (UID: \"1951f4ec-2955-4b93-a40e-34de8400b217\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:55:56.117869 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.117754    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1951f4ec-2955-4b93-a40e-34de8400b217-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f\" (UID: \"1951f4ec-2955-4b93-a40e-34de8400b217\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:55:56.117869 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.117827    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1951f4ec-2955-4b93-a40e-34de8400b217-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f\" (UID: \"1951f4ec-2955-4b93-a40e-34de8400b217\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:55:56.117869 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.117847    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1951f4ec-2955-4b93-a40e-34de8400b217-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f\" (UID: \"1951f4ec-2955-4b93-a40e-34de8400b217\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:55:56.118012 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.117867    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1951f4ec-2955-4b93-a40e-34de8400b217-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f\" (UID: \"1951f4ec-2955-4b93-a40e-34de8400b217\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:55:56.118012 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.117897    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1951f4ec-2955-4b93-a40e-34de8400b217-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f\" (UID: \"1951f4ec-2955-4b93-a40e-34de8400b217\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:55:56.118153 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.118132    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1951f4ec-2955-4b93-a40e-34de8400b217-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f\" (UID: \"1951f4ec-2955-4b93-a40e-34de8400b217\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:55:56.118219 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.118167    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1951f4ec-2955-4b93-a40e-34de8400b217-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f\" (UID: \"1951f4ec-2955-4b93-a40e-34de8400b217\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:55:56.118289 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.118264    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1951f4ec-2955-4b93-a40e-34de8400b217-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f\" (UID: \"1951f4ec-2955-4b93-a40e-34de8400b217\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:55:56.120179 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.120152    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1951f4ec-2955-4b93-a40e-34de8400b217-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f\" (UID: \"1951f4ec-2955-4b93-a40e-34de8400b217\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:55:56.120454 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.120438    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1951f4ec-2955-4b93-a40e-34de8400b217-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f\" (UID: \"1951f4ec-2955-4b93-a40e-34de8400b217\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:55:56.181185 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.181127    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fnxz\" (UniqueName: \"kubernetes.io/projected/1951f4ec-2955-4b93-a40e-34de8400b217-kube-api-access-7fnxz\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f\" (UID: \"1951f4ec-2955-4b93-a40e-34de8400b217\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:55:56.284297 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.284271    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:55:56.661179 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:56.660918    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"]
Nov 01 05:55:56.663766 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:55:56.663738    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1951f4ec_2955_4b93_a40e_34de8400b217.slice/crio-3a483df8b73a1be0e54eebf44180131b5649b2eab2f31b5b1b7641a71b674cae WatchSource:0}: Error finding container 3a483df8b73a1be0e54eebf44180131b5649b2eab2f31b5b1b7641a71b674cae: Status 404 returned error can't find the container with id 3a483df8b73a1be0e54eebf44180131b5649b2eab2f31b5b1b7641a71b674cae
Nov 01 05:55:57.410429 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:57.410392    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f" event={"ID":"1951f4ec-2955-4b93-a40e-34de8400b217","Type":"ContainerStarted","Data":"3a483df8b73a1be0e54eebf44180131b5649b2eab2f31b5b1b7641a71b674cae"}
Nov 01 05:55:59.421169 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:59.421129    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f" event={"ID":"1951f4ec-2955-4b93-a40e-34de8400b217","Type":"ContainerStarted","Data":"dbf6649d5cad6e67c831839af65f086dc309a4fa1aa18c5f6f47246b297f4892"}
Nov 01 05:55:59.540728 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:55:59.540668    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f" podStartSLOduration=2.318071447 podStartE2EDuration="4.540653512s" totalImagesPullingTime="2.222582065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:55:59.540033372 +0000 UTC m=+596.750856881" watchObservedRunningTime="2026-11-01 05:55:59.540653512 +0000 UTC m=+596.751477022"
Nov 01 05:56:00.285036 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:00.285002    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:56:05.055031 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:05.054981    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:06.284688 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:06.284648    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:56:06.297293 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:06.297262    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:56:06.463357 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:06.463317    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f"
Nov 01 05:56:08.253857 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:08.253822    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:11.475410 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.475382    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"]
Nov 01 05:56:11.479016 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.479000    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:11.564576 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.564539    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/305429ba-4531-4538-86e2-34e1a9387083-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jvlvd\" (UID: \"305429ba-4531-4538-86e2-34e1a9387083\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:11.564753 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.564591    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/305429ba-4531-4538-86e2-34e1a9387083-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jvlvd\" (UID: \"305429ba-4531-4538-86e2-34e1a9387083\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:11.564753 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.564613    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56h7k\" (UniqueName: \"kubernetes.io/projected/305429ba-4531-4538-86e2-34e1a9387083-kube-api-access-56h7k\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jvlvd\" (UID: \"305429ba-4531-4538-86e2-34e1a9387083\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:11.564753 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.564696    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/305429ba-4531-4538-86e2-34e1a9387083-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jvlvd\" (UID: \"305429ba-4531-4538-86e2-34e1a9387083\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:11.564889 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.564793    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/305429ba-4531-4538-86e2-34e1a9387083-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jvlvd\" (UID: \"305429ba-4531-4538-86e2-34e1a9387083\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:11.564889 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.564857    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/305429ba-4531-4538-86e2-34e1a9387083-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jvlvd\" (UID: \"305429ba-4531-4538-86e2-34e1a9387083\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:11.572824 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.572794    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:11.591024 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.590999    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 01 05:56:11.592494 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.592473    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"]
Nov 01 05:56:11.665728 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.665695    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/305429ba-4531-4538-86e2-34e1a9387083-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jvlvd\" (UID: \"305429ba-4531-4538-86e2-34e1a9387083\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:11.665728 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.665729    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56h7k\" (UniqueName: \"kubernetes.io/projected/305429ba-4531-4538-86e2-34e1a9387083-kube-api-access-56h7k\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jvlvd\" (UID: \"305429ba-4531-4538-86e2-34e1a9387083\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:11.665961 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.665759    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/305429ba-4531-4538-86e2-34e1a9387083-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jvlvd\" (UID: \"305429ba-4531-4538-86e2-34e1a9387083\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:11.665961 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.665814    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/305429ba-4531-4538-86e2-34e1a9387083-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jvlvd\" (UID: \"305429ba-4531-4538-86e2-34e1a9387083\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:11.665961 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.665834    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/305429ba-4531-4538-86e2-34e1a9387083-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jvlvd\" (UID: \"305429ba-4531-4538-86e2-34e1a9387083\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:11.665961 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.665899    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/305429ba-4531-4538-86e2-34e1a9387083-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jvlvd\" (UID: \"305429ba-4531-4538-86e2-34e1a9387083\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:11.666175 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.666102    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/305429ba-4531-4538-86e2-34e1a9387083-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jvlvd\" (UID: \"305429ba-4531-4538-86e2-34e1a9387083\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:11.666233 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.666188    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/305429ba-4531-4538-86e2-34e1a9387083-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jvlvd\" (UID: \"305429ba-4531-4538-86e2-34e1a9387083\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:11.666397 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.666376    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/305429ba-4531-4538-86e2-34e1a9387083-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jvlvd\" (UID: \"305429ba-4531-4538-86e2-34e1a9387083\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:11.668232 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.668212    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/305429ba-4531-4538-86e2-34e1a9387083-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jvlvd\" (UID: \"305429ba-4531-4538-86e2-34e1a9387083\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:11.668415 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.668398    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/305429ba-4531-4538-86e2-34e1a9387083-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jvlvd\" (UID: \"305429ba-4531-4538-86e2-34e1a9387083\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:11.727555 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.727488    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56h7k\" (UniqueName: \"kubernetes.io/projected/305429ba-4531-4538-86e2-34e1a9387083-kube-api-access-56h7k\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jvlvd\" (UID: \"305429ba-4531-4538-86e2-34e1a9387083\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:11.791124 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:11.791088    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:12.164960 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:12.164712    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"]
Nov 01 05:56:12.167749 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:56:12.167721    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305429ba_4531_4538_86e2_34e1a9387083.slice/crio-07d30fa3ad711f3497fa628ed275b5adf1219096549f7269256faa0990ced28b WatchSource:0}: Error finding container 07d30fa3ad711f3497fa628ed275b5adf1219096549f7269256faa0990ced28b: Status 404 returned error can't find the container with id 07d30fa3ad711f3497fa628ed275b5adf1219096549f7269256faa0990ced28b
Nov 01 05:56:12.169040 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:12.169017    5619 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 05:56:12.482918 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:12.482837    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd" event={"ID":"305429ba-4531-4538-86e2-34e1a9387083","Type":"ContainerStarted","Data":"07d30fa3ad711f3497fa628ed275b5adf1219096549f7269256faa0990ced28b"}
Nov 01 05:56:13.488508 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:13.488468    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd" event={"ID":"305429ba-4531-4538-86e2-34e1a9387083","Type":"ContainerStarted","Data":"52bba8588e158ad756cca4ea78c559a1c74233b00861811b5201b0da7d5eec43"}
Nov 01 05:56:13.608391 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:13.608336    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd" podStartSLOduration=1.841123957 podStartE2EDuration="2.608305895s" totalImagesPullingTime="767.181938ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:56:13.606715191 +0000 UTC m=+610.817538701" watchObservedRunningTime="2026-11-01 05:56:13.608305895 +0000 UTC m=+610.819129403"
Nov 01 05:56:13.792303 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:13.792226    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:14.569645 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:14.569607    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:16.646889 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:16.646855    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:18.430917 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:18.430888    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:18.575922 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:18.575888    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:19.743874 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:19.743842    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-npc9b"]
Nov 01 05:56:19.746748 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:19.746729    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:19.833474 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:19.833418    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:19.840090 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:19.840055    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25955c22-ca47-4d49-8253-0f9660b74633-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-npc9b\" (UID: \"25955c22-ca47-4d49-8253-0f9660b74633\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:19.840090 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:19.840086    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25955c22-ca47-4d49-8253-0f9660b74633-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-npc9b\" (UID: \"25955c22-ca47-4d49-8253-0f9660b74633\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:19.840315 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:19.840220    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25955c22-ca47-4d49-8253-0f9660b74633-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-npc9b\" (UID: \"25955c22-ca47-4d49-8253-0f9660b74633\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:19.840315 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:19.840275    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25955c22-ca47-4d49-8253-0f9660b74633-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-npc9b\" (UID: \"25955c22-ca47-4d49-8253-0f9660b74633\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:19.840431 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:19.840372    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25t6l\" (UniqueName: \"kubernetes.io/projected/25955c22-ca47-4d49-8253-0f9660b74633-kube-api-access-25t6l\") pod \"echo-simulator-kserve-64b6f8f96c-npc9b\" (UID: \"25955c22-ca47-4d49-8253-0f9660b74633\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:19.840431 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:19.840409    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25955c22-ca47-4d49-8253-0f9660b74633-home\") pod \"echo-simulator-kserve-64b6f8f96c-npc9b\" (UID: \"25955c22-ca47-4d49-8253-0f9660b74633\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:19.850682 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:19.850650    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 01 05:56:19.852572 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:19.852555    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-npc9b"]
Nov 01 05:56:19.941249 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:19.941221    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25955c22-ca47-4d49-8253-0f9660b74633-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-npc9b\" (UID: \"25955c22-ca47-4d49-8253-0f9660b74633\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:19.941249 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:19.941252    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25955c22-ca47-4d49-8253-0f9660b74633-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-npc9b\" (UID: \"25955c22-ca47-4d49-8253-0f9660b74633\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:19.941477 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:19.941304    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25955c22-ca47-4d49-8253-0f9660b74633-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-npc9b\" (UID: \"25955c22-ca47-4d49-8253-0f9660b74633\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:19.941477 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:19.941348    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25955c22-ca47-4d49-8253-0f9660b74633-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-npc9b\" (UID: \"25955c22-ca47-4d49-8253-0f9660b74633\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:19.941477 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:19.941379    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25t6l\" (UniqueName: \"kubernetes.io/projected/25955c22-ca47-4d49-8253-0f9660b74633-kube-api-access-25t6l\") pod \"echo-simulator-kserve-64b6f8f96c-npc9b\" (UID: \"25955c22-ca47-4d49-8253-0f9660b74633\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:19.941621 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:19.941522    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25955c22-ca47-4d49-8253-0f9660b74633-home\") pod \"echo-simulator-kserve-64b6f8f96c-npc9b\" (UID: \"25955c22-ca47-4d49-8253-0f9660b74633\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:19.941799 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:19.941763    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25955c22-ca47-4d49-8253-0f9660b74633-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-npc9b\" (UID: \"25955c22-ca47-4d49-8253-0f9660b74633\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:19.941799 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:19.941792    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25955c22-ca47-4d49-8253-0f9660b74633-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-npc9b\" (UID: \"25955c22-ca47-4d49-8253-0f9660b74633\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:19.941987 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:19.941886    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25955c22-ca47-4d49-8253-0f9660b74633-home\") pod \"echo-simulator-kserve-64b6f8f96c-npc9b\" (UID: \"25955c22-ca47-4d49-8253-0f9660b74633\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:19.943513 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:19.943493    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25955c22-ca47-4d49-8253-0f9660b74633-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-npc9b\" (UID: \"25955c22-ca47-4d49-8253-0f9660b74633\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:19.943881 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:19.943861    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25955c22-ca47-4d49-8253-0f9660b74633-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-npc9b\" (UID: \"25955c22-ca47-4d49-8253-0f9660b74633\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:20.002959 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:20.002903    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25t6l\" (UniqueName: \"kubernetes.io/projected/25955c22-ca47-4d49-8253-0f9660b74633-kube-api-access-25t6l\") pod \"echo-simulator-kserve-64b6f8f96c-npc9b\" (UID: \"25955c22-ca47-4d49-8253-0f9660b74633\") " pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:20.057526 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:20.057500    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:20.428153 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:20.428125    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-npc9b"]
Nov 01 05:56:20.430830 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:56:20.430799    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25955c22_ca47_4d49_8253_0f9660b74633.slice/crio-64bd24ef5ed8128c079a0486dc2079ced3caef969781577bdc4e705225aeffb9 WatchSource:0}: Error finding container 64bd24ef5ed8128c079a0486dc2079ced3caef969781577bdc4e705225aeffb9: Status 404 returned error can't find the container with id 64bd24ef5ed8128c079a0486dc2079ced3caef969781577bdc4e705225aeffb9
Nov 01 05:56:20.520946 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:20.520912    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b" event={"ID":"25955c22-ca47-4d49-8253-0f9660b74633","Type":"ContainerStarted","Data":"64bd24ef5ed8128c079a0486dc2079ced3caef969781577bdc4e705225aeffb9"}
Nov 01 05:56:21.791843 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:21.791802    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:21.809240 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:21.809212    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:22.540725 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:22.540695    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jvlvd"
Nov 01 05:56:23.537970 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:23.537937    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b" event={"ID":"25955c22-ca47-4d49-8253-0f9660b74633","Type":"ContainerStarted","Data":"7d43afde4ac094e237875e60bae1b5585648604ba6206cc56f08c48a518b3a98"}
Nov 01 05:56:23.656768 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:23.656709    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b" podStartSLOduration=2.105590583 podStartE2EDuration="4.656691761s" totalImagesPullingTime="2.551101178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:56:23.656489766 +0000 UTC m=+620.867313288" watchObservedRunningTime="2026-11-01 05:56:23.656691761 +0000 UTC m=+620.867515270"
Nov 01 05:56:24.057834 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:24.057804    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:24.309417 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:24.309343    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:25.881478 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:56:25.881443    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:56:28.655294 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:28.655257    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:30.057948 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:30.057912    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:30.070663 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:30.070639    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:30.584911 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:30.584882    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-npc9b"
Nov 01 05:56:53.260097 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.260026    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"]
Nov 01 05:56:53.263692 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.263672    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.317378 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.317346    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vqwlh\""
Nov 01 05:56:53.323621 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.323592    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmghx\" (UniqueName: \"kubernetes.io/projected/d299e7a4-892c-404f-8346-91bcae60326e-kube-api-access-pmghx\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.323735 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.323660    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d299e7a4-892c-404f-8346-91bcae60326e-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.323787 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.323735    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d299e7a4-892c-404f-8346-91bcae60326e-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.323787 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.323763    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d299e7a4-892c-404f-8346-91bcae60326e-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.323878 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.323849    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d299e7a4-892c-404f-8346-91bcae60326e-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.323931 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.323876    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d299e7a4-892c-404f-8346-91bcae60326e-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.323931 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.323908    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d299e7a4-892c-404f-8346-91bcae60326e-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.324037 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.323971    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d299e7a4-892c-404f-8346-91bcae60326e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.324088 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.324035    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d299e7a4-892c-404f-8346-91bcae60326e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.371267 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.371235    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"]
Nov 01 05:56:53.425461 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.425419    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d299e7a4-892c-404f-8346-91bcae60326e-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.425653 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.425474    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d299e7a4-892c-404f-8346-91bcae60326e-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.425653 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.425502    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d299e7a4-892c-404f-8346-91bcae60326e-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.425653 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.425559    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d299e7a4-892c-404f-8346-91bcae60326e-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.425653 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.425598    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d299e7a4-892c-404f-8346-91bcae60326e-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.425868 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.425668    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d299e7a4-892c-404f-8346-91bcae60326e-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.425868 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.425704    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d299e7a4-892c-404f-8346-91bcae60326e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.425868 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.425737    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d299e7a4-892c-404f-8346-91bcae60326e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.425868 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.425779    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmghx\" (UniqueName: \"kubernetes.io/projected/d299e7a4-892c-404f-8346-91bcae60326e-kube-api-access-pmghx\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.426084 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.425900    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d299e7a4-892c-404f-8346-91bcae60326e-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.426084 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.425944    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d299e7a4-892c-404f-8346-91bcae60326e-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.426197 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.426093    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d299e7a4-892c-404f-8346-91bcae60326e-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.426197 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.426119    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d299e7a4-892c-404f-8346-91bcae60326e-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.426557 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.426537    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d299e7a4-892c-404f-8346-91bcae60326e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.428095 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.428051    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d299e7a4-892c-404f-8346-91bcae60326e-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.428362 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.428314    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d299e7a4-892c-404f-8346-91bcae60326e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.487861 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.487829    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d299e7a4-892c-404f-8346-91bcae60326e-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.487997 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.487844    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmghx\" (UniqueName: \"kubernetes.io/projected/d299e7a4-892c-404f-8346-91bcae60326e-kube-api-access-pmghx\") pod \"maas-default-gateway-openshift-default-855f9574cf-x9mzj\" (UID: \"d299e7a4-892c-404f-8346-91bcae60326e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:53.576965 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:53.576886    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:54.005365 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:54.005310    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"]
Nov 01 05:56:54.008045 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:56:54.008018    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd299e7a4_892c_404f_8346_91bcae60326e.slice/crio-96895fabd7c85b330d2139e4f503d21c03c89bac420f490c423470e715b3914a WatchSource:0}: Error finding container 96895fabd7c85b330d2139e4f503d21c03c89bac420f490c423470e715b3914a: Status 404 returned error can't find the container with id 96895fabd7c85b330d2139e4f503d21c03c89bac420f490c423470e715b3914a
Nov 01 05:56:54.010107 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:54.010077    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:56:54.010161 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:54.010140    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:56:54.010205 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:54.010170    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:56:54.682160 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:54.682115    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj" event={"ID":"d299e7a4-892c-404f-8346-91bcae60326e","Type":"ContainerStarted","Data":"b7b9a2cbb7c623ad3b17001399fd19700a27dcde12f29136bd29555f7bc74347"}
Nov 01 05:56:54.682630 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:54.682229    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj" event={"ID":"d299e7a4-892c-404f-8346-91bcae60326e","Type":"ContainerStarted","Data":"96895fabd7c85b330d2139e4f503d21c03c89bac420f490c423470e715b3914a"}
Nov 01 05:56:55.577890 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:55.577848    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:56:55.579596 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:55.579562    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-x9mzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 01 05:56:55.579720 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:55.579617    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj" podUID="d299e7a4-892c-404f-8346-91bcae60326e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 01 05:56:55.897910 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:56:55.897874    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:56:56.577592 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:56.577549    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-x9mzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 01 05:56:56.577771 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:56.577613    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj" podUID="d299e7a4-892c-404f-8346-91bcae60326e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 01 05:56:57.577614 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:57.577570    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-x9mzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 01 05:56:57.578016 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:57.577648    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj" podUID="d299e7a4-892c-404f-8346-91bcae60326e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 01 05:56:58.578102 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:58.578069    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-x9mzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 01 05:56:58.578598 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:58.578136    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj" podUID="d299e7a4-892c-404f-8346-91bcae60326e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 01 05:56:59.578114 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:59.578079    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-x9mzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 01 05:56:59.578628 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:59.578145    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj" podUID="d299e7a4-892c-404f-8346-91bcae60326e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 01 05:56:59.720772 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:59.720739    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:56:59.870850 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:56:59.870816    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:00.577407 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:00.577361    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-x9mzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 01 05:57:00.577597 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:00.577461    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj" podUID="d299e7a4-892c-404f-8346-91bcae60326e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 01 05:57:01.289099 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:01.289067    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:01.429862 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:01.429816    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:02.299655 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:02.299626    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:57:02.300221 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:02.300193    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:57:02.301103 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:02.301078    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj"
Nov 01 05:57:02.524643 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:02.524587    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj" podStartSLOduration=9.52457177 podStartE2EDuration="9.52457177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:56:54.870566225 +0000 UTC m=+652.081389746" watchObservedRunningTime="2026-11-01 05:57:02.52457177 +0000 UTC m=+659.735395350"
Nov 01 05:57:08.380674 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.380641    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"]
Nov 01 05:57:08.385494 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.385477    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.491606 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.491577    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"]
Nov 01 05:57:08.493136 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.493107    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.493273 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.493157    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8ndb\" (UniqueName: \"kubernetes.io/projected/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-kube-api-access-v8ndb\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.493334 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.493296    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.493404 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.493386    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.493459 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.493442    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.493521 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.493478    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.493577 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.493515    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.493577 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.493544    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.493685 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.493612    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.594504 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.594473    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.595138 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.594525    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.595138 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.594552    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.595138 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.594583    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.595138 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.594640    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.595138 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.594668    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.595138 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.594724    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.595138 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.594765    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.595138 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.594811    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8ndb\" (UniqueName: \"kubernetes.io/projected/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-kube-api-access-v8ndb\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.595138 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.594922    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.595138 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.594965    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.595138 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.595044    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.595138 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.595137    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.595841 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.595670    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.597348 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.597304    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.597656 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.597629    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.656756 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.656702    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.656926 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.656903    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8ndb\" (UniqueName: \"kubernetes.io/projected/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-kube-api-access-v8ndb\") pod \"maas-default-gateway-openshift-default-855f9574cf-c8mt2\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:08.698168 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:08.698146    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:09.124863 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:09.124831    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"]
Nov 01 05:57:09.128098 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:57:09.128069    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2670ba1b_cdaa_48f4_9808_b34fa5632cbd.slice/crio-4a07e82085e8938bc628d5ceef06e6545ec8e0e5095010ae61eead89158e84ee WatchSource:0}: Error finding container 4a07e82085e8938bc628d5ceef06e6545ec8e0e5095010ae61eead89158e84ee: Status 404 returned error can't find the container with id 4a07e82085e8938bc628d5ceef06e6545ec8e0e5095010ae61eead89158e84ee
Nov 01 05:57:09.130753 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:09.130718    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:57:09.137781 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:09.131176    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:57:09.137781 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:09.131298    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:57:09.771348 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:09.764921    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2" event={"ID":"2670ba1b-cdaa-48f4-9808-b34fa5632cbd","Type":"ContainerStarted","Data":"a082f75ccc48904e6c02e06c26a00f14db4c5918ee6874e26a2c97e2dcb43f08"}
Nov 01 05:57:09.771348 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:09.765066    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2" event={"ID":"2670ba1b-cdaa-48f4-9808-b34fa5632cbd","Type":"ContainerStarted","Data":"4a07e82085e8938bc628d5ceef06e6545ec8e0e5095010ae61eead89158e84ee"}
Nov 01 05:57:09.888119 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:09.888065    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2" podStartSLOduration=1.8880503499999999 podStartE2EDuration="1.88805035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:57:09.886584005 +0000 UTC m=+667.097407516" watchObservedRunningTime="2026-11-01 05:57:09.88805035 +0000 UTC m=+667.098873859"
Nov 01 05:57:10.698762 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:10.698721    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:10.700417 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:10.700389    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-c8mt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 01 05:57:10.700553 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:10.700437    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2" podUID="2670ba1b-cdaa-48f4-9808-b34fa5632cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 01 05:57:11.699462 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:11.699428    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-c8mt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 01 05:57:11.699852 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:11.699485    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2" podUID="2670ba1b-cdaa-48f4-9808-b34fa5632cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 01 05:57:12.699526 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:12.699492    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-c8mt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 01 05:57:12.699921 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:12.699546    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2" podUID="2670ba1b-cdaa-48f4-9808-b34fa5632cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 01 05:57:13.698691 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:13.698656    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-c8mt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 01 05:57:13.698883 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:13.698723    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2" podUID="2670ba1b-cdaa-48f4-9808-b34fa5632cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 01 05:57:14.699101 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:14.699065    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-c8mt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 01 05:57:14.699530 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:14.699121    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2" podUID="2670ba1b-cdaa-48f4-9808-b34fa5632cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 01 05:57:14.957403 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:14.957290    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:15.699455 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:15.699407    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-c8mt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 01 05:57:15.699866 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:15.699485    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2" podUID="2670ba1b-cdaa-48f4-9808-b34fa5632cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 01 05:57:17.335803 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:17.335774    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:17.336287 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:17.336066    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:17.336867 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:17.336847    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 05:57:25.878264 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:57:25.878228    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:57:42.213588 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:42.213543    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:47.316524 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:47.316487    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:47.511034 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:47.510979    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:50.226442 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:50.226407    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:50.365531 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:50.365494    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:50.852682 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:50.852644    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:51.068667 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.068628    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:51.069518 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.069488    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"]
Nov 01 05:57:51.072973 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.072953    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.127058 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.126965    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-f73e89-openshift-default-dockercfg-tqlnq\""
Nov 01 05:57:51.134849 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.134818    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"]
Nov 01 05:57:51.189566 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.189536    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48e404e-08bf-4a58-84f8-3c095893759f-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.189759 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.189578    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trmbj\" (UniqueName: \"kubernetes.io/projected/b48e404e-08bf-4a58-84f8-3c095893759f-kube-api-access-trmbj\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.189759 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.189617    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.189759 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.189713    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.189759 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.189744    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.189979 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.189768    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.189979 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.189808    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48e404e-08bf-4a58-84f8-3c095893759f-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.189979 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.189887    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48e404e-08bf-4a58-84f8-3c095893759f-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.189979 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.189913    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.218265 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.218221    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:51.290688 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.290648    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48e404e-08bf-4a58-84f8-3c095893759f-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.291162 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.290694    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trmbj\" (UniqueName: \"kubernetes.io/projected/b48e404e-08bf-4a58-84f8-3c095893759f-kube-api-access-trmbj\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.291162 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.290743    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.291162 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.290789    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.291162 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.290819    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.291162 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.290849    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.291162 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.290873    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48e404e-08bf-4a58-84f8-3c095893759f-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.291162 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.290925    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48e404e-08bf-4a58-84f8-3c095893759f-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.291162 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.290967    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.291651 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.291290    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.291651 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.291365    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.291651 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.291532    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.291807 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.291661    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.291807 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.291675    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48e404e-08bf-4a58-84f8-3c095893759f-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.293833 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.293808    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.293962 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.293921    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48e404e-08bf-4a58-84f8-3c095893759f-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.352481 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.352455    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trmbj\" (UniqueName: \"kubernetes.io/projected/b48e404e-08bf-4a58-84f8-3c095893759f-kube-api-access-trmbj\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.352673 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.352656    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48e404e-08bf-4a58-84f8-3c095893759f-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.385640 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.385585    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:51.502684 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.502650    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:51.819863 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.819793    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"]
Nov 01 05:57:51.822769 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:57:51.822742    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb48e404e_08bf_4a58_84f8_3c095893759f.slice/crio-5394445e8166ac89345398e78e10c0c041f56583b3c35b374f67de009eb9d19f WatchSource:0}: Error finding container 5394445e8166ac89345398e78e10c0c041f56583b3c35b374f67de009eb9d19f: Status 404 returned error can't find the container with id 5394445e8166ac89345398e78e10c0c041f56583b3c35b374f67de009eb9d19f
Nov 01 05:57:51.824938 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.824907    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:57:51.825037 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.824976    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:57:51.825037 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.825005    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:57:51.971447 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.971406    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl" event={"ID":"b48e404e-08bf-4a58-84f8-3c095893759f","Type":"ContainerStarted","Data":"de2c553c27c69ac68e6669383e58d9ea1eeff584ccf94154de9d756a5c592682"}
Nov 01 05:57:51.971607 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:51.971469    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl" event={"ID":"b48e404e-08bf-4a58-84f8-3c095893759f","Type":"ContainerStarted","Data":"5394445e8166ac89345398e78e10c0c041f56583b3c35b374f67de009eb9d19f"}
Nov 01 05:57:52.095702 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:52.095641    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl" podStartSLOduration=1.095621059 podStartE2EDuration="1.095621059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:57:52.092909034 +0000 UTC m=+709.303732542" watchObservedRunningTime="2026-11-01 05:57:52.095621059 +0000 UTC m=+709.306444568"
Nov 01 05:57:52.385871 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:52.385796    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:52.390615 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:52.390592    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:52.975634 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:52.975599    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:52.976599 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:52.976580    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:57:55.035447 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:55.035411    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:55.289195 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:55.289078    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:55.667804 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:55.667770    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:55.881011 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:57:55.880972    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:57:55.888835 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:55.888788    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:56.037296 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:56.037212    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:58.245961 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:58.245928    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:57:58.428351 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:57:58.428284    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:00.305851 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:00.305816    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:00.454946 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:00.454911    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:00.871597 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:00.871562    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:01.066023 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:01.065980    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:01.466239 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:01.466210    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"]
Nov 01 05:58:01.474376 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:01.474350    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"
Nov 01 05:58:01.527410 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:01.527378    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-9c8fc5-serving-cert\""
Nov 01 05:58:01.530060 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:01.530025    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"]
Nov 01 05:58:01.599431 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:01.599401    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b960f047-0688-4d8d-998b-9bc0f5b595b1-metrics-certs\") pod \"maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s\" (UID: \"b960f047-0688-4d8d-998b-9bc0f5b595b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"
Nov 01 05:58:01.599631 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:01.599461    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5zk2\" (UniqueName: \"kubernetes.io/projected/b960f047-0688-4d8d-998b-9bc0f5b595b1-kube-api-access-p5zk2\") pod \"maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s\" (UID: \"b960f047-0688-4d8d-998b-9bc0f5b595b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"
Nov 01 05:58:01.599631 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:01.599590    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b960f047-0688-4d8d-998b-9bc0f5b595b1-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s\" (UID: \"b960f047-0688-4d8d-998b-9bc0f5b595b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"
Nov 01 05:58:01.599728 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:01.599646    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b960f047-0688-4d8d-998b-9bc0f5b595b1-maas-api-tls\") pod \"maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s\" (UID: \"b960f047-0688-4d8d-998b-9bc0f5b595b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"
Nov 01 05:58:01.660661 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:01.660625    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:01.700778 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:01.700746    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b960f047-0688-4d8d-998b-9bc0f5b595b1-metrics-certs\") pod \"maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s\" (UID: \"b960f047-0688-4d8d-998b-9bc0f5b595b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"
Nov 01 05:58:01.700967 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:01.700802    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5zk2\" (UniqueName: \"kubernetes.io/projected/b960f047-0688-4d8d-998b-9bc0f5b595b1-kube-api-access-p5zk2\") pod \"maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s\" (UID: \"b960f047-0688-4d8d-998b-9bc0f5b595b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"
Nov 01 05:58:01.700967 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:01.700848    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b960f047-0688-4d8d-998b-9bc0f5b595b1-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s\" (UID: \"b960f047-0688-4d8d-998b-9bc0f5b595b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"
Nov 01 05:58:01.700967 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:01.700882    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b960f047-0688-4d8d-998b-9bc0f5b595b1-maas-api-tls\") pod \"maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s\" (UID: \"b960f047-0688-4d8d-998b-9bc0f5b595b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"
Nov 01 05:58:01.701136 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:58:01.701059    5619 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-serving-cert: secret "maas-api-e2e-worker-w0-9c8fc5-serving-cert" not found
Nov 01 05:58:01.701185 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:58:01.701151    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b960f047-0688-4d8d-998b-9bc0f5b595b1-maas-api-tls podName:b960f047-0688-4d8d-998b-9bc0f5b595b1 nodeName:}" failed. No retries permitted until 2026-11-01 05:58:02.201128674 +0000 UTC m=+719.411952175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b960f047-0688-4d8d-998b-9bc0f5b595b1-maas-api-tls") pod "maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s" (UID: "b960f047-0688-4d8d-998b-9bc0f5b595b1") : secret "maas-api-e2e-worker-w0-9c8fc5-serving-cert" not found
Nov 01 05:58:01.701605 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:01.701585    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b960f047-0688-4d8d-998b-9bc0f5b595b1-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s\" (UID: \"b960f047-0688-4d8d-998b-9bc0f5b595b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"
Nov 01 05:58:01.703144 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:01.703126    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b960f047-0688-4d8d-998b-9bc0f5b595b1-metrics-certs\") pod \"maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s\" (UID: \"b960f047-0688-4d8d-998b-9bc0f5b595b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"
Nov 01 05:58:01.765183 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:01.765105    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5zk2\" (UniqueName: \"kubernetes.io/projected/b960f047-0688-4d8d-998b-9bc0f5b595b1-kube-api-access-p5zk2\") pod \"maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s\" (UID: \"b960f047-0688-4d8d-998b-9bc0f5b595b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"
Nov 01 05:58:01.820966 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:01.820938    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:01.969063 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:01.969026    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:02.205772 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:02.205736    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b960f047-0688-4d8d-998b-9bc0f5b595b1-maas-api-tls\") pod \"maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s\" (UID: \"b960f047-0688-4d8d-998b-9bc0f5b595b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"
Nov 01 05:58:02.208181 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:02.208155    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b960f047-0688-4d8d-998b-9bc0f5b595b1-maas-api-tls\") pod \"maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s\" (UID: \"b960f047-0688-4d8d-998b-9bc0f5b595b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"
Nov 01 05:58:02.390237 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:02.390202    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"
Nov 01 05:58:02.767388 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:02.767355    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"]
Nov 01 05:58:02.769592 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:58:02.769565    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb960f047_0688_4d8d_998b_9bc0f5b595b1.slice/crio-a7c5b69103055fed6bb69910a6e79ee2c47cd0c790ecf6c3a54d9ae06db2020f WatchSource:0}: Error finding container a7c5b69103055fed6bb69910a6e79ee2c47cd0c790ecf6c3a54d9ae06db2020f: Status 404 returned error can't find the container with id a7c5b69103055fed6bb69910a6e79ee2c47cd0c790ecf6c3a54d9ae06db2020f
Nov 01 05:58:02.999007 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:02.998969    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:03.024101 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:03.024019    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s" event={"ID":"b960f047-0688-4d8d-998b-9bc0f5b595b1","Type":"ContainerStarted","Data":"a7c5b69103055fed6bb69910a6e79ee2c47cd0c790ecf6c3a54d9ae06db2020f"}
Nov 01 05:58:03.178445 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:03.178409    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:04.031502 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:04.031421    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s" event={"ID":"b960f047-0688-4d8d-998b-9bc0f5b595b1","Type":"ContainerStarted","Data":"8b2fb0045a11ae53db06ee12ed39d74f80ba71ce38f96a589f374321087d66e1"}
Nov 01 05:58:04.031859 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:04.031556    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"
Nov 01 05:58:04.151929 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:04.151874    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s" podStartSLOduration=2.300107151 podStartE2EDuration="3.151859139s" totalImagesPullingTime="851.751988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:04.149945596 +0000 UTC m=+721.360769104" watchObservedRunningTime="2026-11-01 05:58:04.151859139 +0000 UTC m=+721.362682647"
Nov 01 05:58:06.315432 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:06.315399    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"]
Nov 01 05:58:06.319256 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:06.319237    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"
Nov 01 05:58:06.372490 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:06.372459    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-f73e89-serving-cert\""
Nov 01 05:58:06.426107 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:06.426075    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"]
Nov 01 05:58:06.448601 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:06.448577    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m\" (UID: \"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"
Nov 01 05:58:06.448744 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:06.448719    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-metrics-certs\") pod \"maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m\" (UID: \"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"
Nov 01 05:58:06.448803 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:06.448762    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5c9g\" (UniqueName: \"kubernetes.io/projected/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-kube-api-access-j5c9g\") pod \"maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m\" (UID: \"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"
Nov 01 05:58:06.448803 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:06.448790    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m\" (UID: \"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"
Nov 01 05:58:06.516245 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:06.516213    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:06.549871 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:06.549845    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-metrics-certs\") pod \"maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m\" (UID: \"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"
Nov 01 05:58:06.550019 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:06.549884    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5c9g\" (UniqueName: \"kubernetes.io/projected/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-kube-api-access-j5c9g\") pod \"maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m\" (UID: \"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"
Nov 01 05:58:06.550019 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:06.549909    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m\" (UID: \"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"
Nov 01 05:58:06.550019 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:06.549951    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m\" (UID: \"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"
Nov 01 05:58:06.550196 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:58:06.550023    5619 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-serving-cert: secret "maas-api-e2e-worker-w4-f73e89-serving-cert" not found
Nov 01 05:58:06.550196 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:58:06.550093    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-maas-api-tls podName:d7e166f7-3bf1-4d53-b9e8-e964a5546a8d nodeName:}" failed. No retries permitted until 2026-11-01 05:58:07.050074969 +0000 UTC m=+724.260898484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-maas-api-tls") pod "maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m" (UID: "d7e166f7-3bf1-4d53-b9e8-e964a5546a8d") : secret "maas-api-e2e-worker-w4-f73e89-serving-cert" not found
Nov 01 05:58:06.550667 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:06.550647    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m\" (UID: \"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"
Nov 01 05:58:06.552356 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:06.552311    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-metrics-certs\") pod \"maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m\" (UID: \"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"
Nov 01 05:58:06.614159 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:06.614129    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5c9g\" (UniqueName: \"kubernetes.io/projected/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-kube-api-access-j5c9g\") pod \"maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m\" (UID: \"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"
Nov 01 05:58:06.677649 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:06.677617    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:06.822949 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:06.822918    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:07.055132 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:07.055052    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m\" (UID: \"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"
Nov 01 05:58:07.057555 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:07.057533    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m\" (UID: \"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"
Nov 01 05:58:07.231782 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:07.231743    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"
Nov 01 05:58:07.608121 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:07.608098    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"]
Nov 01 05:58:07.610996 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:58:07.610963    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e166f7_3bf1_4d53_b9e8_e964a5546a8d.slice/crio-da3b54e14110d484284025f9bb05c10cdcc7a7cf95adb379996254063f76ec87 WatchSource:0}: Error finding container da3b54e14110d484284025f9bb05c10cdcc7a7cf95adb379996254063f76ec87: Status 404 returned error can't find the container with id da3b54e14110d484284025f9bb05c10cdcc7a7cf95adb379996254063f76ec87
Nov 01 05:58:08.050986 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:08.050890    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m" event={"ID":"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d","Type":"ContainerStarted","Data":"da3b54e14110d484284025f9bb05c10cdcc7a7cf95adb379996254063f76ec87"}
Nov 01 05:58:09.057267 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:09.057231    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m" event={"ID":"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d","Type":"ContainerStarted","Data":"aacf1f51772e84e5b2b0353a1521470408a11a830a540c00765118de582a1fcc"}
Nov 01 05:58:09.057667 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:09.057356    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"
Nov 01 05:58:09.177654 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:09.177609    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m" podStartSLOduration=2.38029069 podStartE2EDuration="3.177596031s" totalImagesPullingTime="797.305341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:09.176714364 +0000 UTC m=+726.387537873" watchObservedRunningTime="2026-11-01 05:58:09.177596031 +0000 UTC m=+726.388419539"
Nov 01 05:58:10.041391 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:10.041362    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"
Nov 01 05:58:11.373548 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:11.373469    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:15.076353 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:15.076303    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"
Nov 01 05:58:15.197124 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:15.197080    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:15.396116 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:15.396076    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:16.662406 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:16.662373    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:18.445762 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:18.445727    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:18.616230 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:18.616185    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:21.982914 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:21.982871    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:23.081963 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:23.081381    5619 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": read tcp 10.133.0.2:37786->10.133.0.38:15021: read: connection reset by peer" start-of-body=
Nov 01 05:58:23.081963 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:23.081441    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl" podUID="b48e404e-08bf-4a58-84f8-3c095893759f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": read tcp 10.133.0.2:37786->10.133.0.38:15021: read: connection reset by peer"
Nov 01 05:58:23.127971 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:23.127943    5619 generic.go:356] "Generic (PLEG): container finished" podID="b48e404e-08bf-4a58-84f8-3c095893759f" containerID="de2c553c27c69ac68e6669383e58d9ea1eeff584ccf94154de9d756a5c592682" exitCode=0
Nov 01 05:58:23.128070 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:23.128014    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl" event={"ID":"b48e404e-08bf-4a58-84f8-3c095893759f","Type":"ContainerDied","Data":"de2c553c27c69ac68e6669383e58d9ea1eeff584ccf94154de9d756a5c592682"}
Nov 01 05:58:23.128391 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:23.128378    5619 scope.go:122] "RemoveContainer" containerID="de2c553c27c69ac68e6669383e58d9ea1eeff584ccf94154de9d756a5c592682"
Nov 01 05:58:23.129534 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:23.129508    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:58:23.129608 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:23.129595    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:58:23.129645 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:23.129629    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:58:24.135155 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:24.135107    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl" event={"ID":"b48e404e-08bf-4a58-84f8-3c095893759f","Type":"ContainerStarted","Data":"9af6a30638ce8f56d68891aaf143bce408c30af45a6be17ca035bf317abd3320"}
Nov 01 05:58:24.386820 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:24.386729    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:58:24.387028 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:24.387004    5619 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 01 05:58:24.387073 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:24.387052    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl" podUID="b48e404e-08bf-4a58-84f8-3c095893759f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 01 05:58:25.188798 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:25.188767    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:25.386057 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:25.386020    5619 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 01 05:58:25.386233 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:25.386070    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl" podUID="b48e404e-08bf-4a58-84f8-3c095893759f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 01 05:58:25.831492 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:25.831444    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:25.899916 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:58:25.899883    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:58:25.982471 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:25.982438    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:26.135684 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:26.135647    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:26.386108 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:26.386035    5619 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 01 05:58:26.386108 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:26.386088    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl" podUID="b48e404e-08bf-4a58-84f8-3c095893759f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 01 05:58:27.262347 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:27.261397    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:27.386041 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:27.386005    5619 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 01 05:58:27.386240 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:27.386065    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl" podUID="b48e404e-08bf-4a58-84f8-3c095893759f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 01 05:58:28.386960 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:28.386917    5619 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 01 05:58:28.387466 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:28.386977    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl" podUID="b48e404e-08bf-4a58-84f8-3c095893759f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 01 05:58:28.536458 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:28.536422    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:29.403376 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:29.403304    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:58:29.404083 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:29.404065    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:58:29.404960 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:29.404903    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 05:58:30.276939 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:30.276903    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:30.866475 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:30.866441    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:31.012993 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:31.012959    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:31.183491 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:31.183388    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:32.303046 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:32.303016    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"]
Nov 01 05:58:32.313478 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:32.313439    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"
Nov 01 05:58:32.352384 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:32.352353    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fa0499e-e650-4321-9162-ca5b191f1ec5-maas-api-tls\") pod \"maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5\" (UID: \"5fa0499e-e650-4321-9162-ca5b191f1ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"
Nov 01 05:58:32.352535 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:32.352410    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fa0499e-e650-4321-9162-ca5b191f1ec5-metrics-certs\") pod \"maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5\" (UID: \"5fa0499e-e650-4321-9162-ca5b191f1ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"
Nov 01 05:58:32.352577 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:32.352546    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa0499e-e650-4321-9162-ca5b191f1ec5-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5\" (UID: \"5fa0499e-e650-4321-9162-ca5b191f1ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"
Nov 01 05:58:32.352618 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:32.352578    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x59d2\" (UniqueName: \"kubernetes.io/projected/5fa0499e-e650-4321-9162-ca5b191f1ec5-kube-api-access-x59d2\") pod \"maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5\" (UID: \"5fa0499e-e650-4321-9162-ca5b191f1ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"
Nov 01 05:58:32.366676 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:32.366646    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"]
Nov 01 05:58:32.414918 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:32.414887    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-14a6e4-serving-cert\""
Nov 01 05:58:32.453188 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:32.453150    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fa0499e-e650-4321-9162-ca5b191f1ec5-maas-api-tls\") pod \"maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5\" (UID: \"5fa0499e-e650-4321-9162-ca5b191f1ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"
Nov 01 05:58:32.453413 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:32.453234    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fa0499e-e650-4321-9162-ca5b191f1ec5-metrics-certs\") pod \"maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5\" (UID: \"5fa0499e-e650-4321-9162-ca5b191f1ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"
Nov 01 05:58:32.453413 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:58:32.453293    5619 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-serving-cert: secret "maas-api-e2e-worker-w2-14a6e4-serving-cert" not found
Nov 01 05:58:32.453413 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:32.453369    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa0499e-e650-4321-9162-ca5b191f1ec5-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5\" (UID: \"5fa0499e-e650-4321-9162-ca5b191f1ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"
Nov 01 05:58:32.453413 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:58:32.453383    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fa0499e-e650-4321-9162-ca5b191f1ec5-maas-api-tls podName:5fa0499e-e650-4321-9162-ca5b191f1ec5 nodeName:}" failed. No retries permitted until 2026-11-01 05:58:32.953360966 +0000 UTC m=+750.164184472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5fa0499e-e650-4321-9162-ca5b191f1ec5-maas-api-tls") pod "maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5" (UID: "5fa0499e-e650-4321-9162-ca5b191f1ec5") : secret "maas-api-e2e-worker-w2-14a6e4-serving-cert" not found
Nov 01 05:58:32.453648 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:32.453425    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x59d2\" (UniqueName: \"kubernetes.io/projected/5fa0499e-e650-4321-9162-ca5b191f1ec5-kube-api-access-x59d2\") pod \"maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5\" (UID: \"5fa0499e-e650-4321-9162-ca5b191f1ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"
Nov 01 05:58:32.454235 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:32.454204    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa0499e-e650-4321-9162-ca5b191f1ec5-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5\" (UID: \"5fa0499e-e650-4321-9162-ca5b191f1ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"
Nov 01 05:58:32.456344 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:32.456292    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fa0499e-e650-4321-9162-ca5b191f1ec5-metrics-certs\") pod \"maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5\" (UID: \"5fa0499e-e650-4321-9162-ca5b191f1ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"
Nov 01 05:58:32.503616 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:32.503587    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:32.517981 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:32.517953    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x59d2\" (UniqueName: \"kubernetes.io/projected/5fa0499e-e650-4321-9162-ca5b191f1ec5-kube-api-access-x59d2\") pod \"maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5\" (UID: \"5fa0499e-e650-4321-9162-ca5b191f1ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"
Nov 01 05:58:32.655242 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:32.655187    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:32.855174 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:32.855138    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:32.957888 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:32.957736    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fa0499e-e650-4321-9162-ca5b191f1ec5-maas-api-tls\") pod \"maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5\" (UID: \"5fa0499e-e650-4321-9162-ca5b191f1ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"
Nov 01 05:58:32.960609 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:32.960578    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fa0499e-e650-4321-9162-ca5b191f1ec5-maas-api-tls\") pod \"maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5\" (UID: \"5fa0499e-e650-4321-9162-ca5b191f1ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"
Nov 01 05:58:33.227905 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:33.227816    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"
Nov 01 05:58:33.603830 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:33.603804    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"]
Nov 01 05:58:33.605912 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:58:33.605882    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa0499e_e650_4321_9162_ca5b191f1ec5.slice/crio-a9a82d003ef98a540b3fb4740ed9565351696301fe8313b78359619221829007 WatchSource:0}: Error finding container a9a82d003ef98a540b3fb4740ed9565351696301fe8313b78359619221829007: Status 404 returned error can't find the container with id a9a82d003ef98a540b3fb4740ed9565351696301fe8313b78359619221829007
Nov 01 05:58:34.214766 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:34.214730    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5" event={"ID":"5fa0499e-e650-4321-9162-ca5b191f1ec5","Type":"ContainerStarted","Data":"a9a82d003ef98a540b3fb4740ed9565351696301fe8313b78359619221829007"}
Nov 01 05:58:35.220567 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:35.220534    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5" event={"ID":"5fa0499e-e650-4321-9162-ca5b191f1ec5","Type":"ContainerStarted","Data":"2522a50ca5f1d2d8561dbd29b50c2f31a98d6defd105298f5e8fcd7eea4a0931"}
Nov 01 05:58:35.220960 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:35.220640    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"
Nov 01 05:58:35.341364 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:35.341293    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5" podStartSLOduration=2.459284201 podStartE2EDuration="3.341277335s" totalImagesPullingTime="881.993134ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:35.339661207 +0000 UTC m=+752.550484718" watchObservedRunningTime="2026-11-01 05:58:35.341277335 +0000 UTC m=+752.552100838"
Nov 01 05:58:37.467213 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:37.467179    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:41.230997 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:41.230966    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"
Nov 01 05:58:42.170304 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:42.170261    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:46.880004 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:46.879960    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:51.776664 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:51.776625    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:53.384319 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.384283    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"]
Nov 01 05:58:53.388007 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.387987    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:58:53.441524 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.441492    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f73e89\"/\"openshift-service-ca.crt\""
Nov 01 05:58:53.441680 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.441533    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f73e89\"/\"kube-root-ca.crt\""
Nov 01 05:58:53.462557 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.462525    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-model-cache\") pod \"simulator-w4-f73e89-kserve-775bd847dc-qz5h2\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") " pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:58:53.462709 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.462566    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-tls-certs\") pod \"simulator-w4-f73e89-kserve-775bd847dc-qz5h2\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") " pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:58:53.462780 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.462699    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grh6s\" (UniqueName: \"kubernetes.io/projected/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-kube-api-access-grh6s\") pod \"simulator-w4-f73e89-kserve-775bd847dc-qz5h2\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") " pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:58:53.462780 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.462764    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-home\") pod \"simulator-w4-f73e89-kserve-775bd847dc-qz5h2\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") " pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:58:53.462865 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.462850    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-dshm\") pod \"simulator-w4-f73e89-kserve-775bd847dc-qz5h2\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") " pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:58:53.462899 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.462874    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-tmp-dir\") pod \"simulator-w4-f73e89-kserve-775bd847dc-qz5h2\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") " pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:58:53.486475 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.486435    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:53.492093 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.492077    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f73e89\"/\"default-dockercfg-262xs\""
Nov 01 05:58:53.492093 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.492088    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f73e89\"/\"simulator-w4-f73e89-kserve-self-signed-certs\""
Nov 01 05:58:53.494065 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.494046    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"]
Nov 01 05:58:53.564485 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.564452    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-model-cache\") pod \"simulator-w4-f73e89-kserve-775bd847dc-qz5h2\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") " pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:58:53.564678 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.564493    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-tls-certs\") pod \"simulator-w4-f73e89-kserve-775bd847dc-qz5h2\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") " pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:58:53.564678 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.564551    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grh6s\" (UniqueName: \"kubernetes.io/projected/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-kube-api-access-grh6s\") pod \"simulator-w4-f73e89-kserve-775bd847dc-qz5h2\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") " pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:58:53.564678 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.564600    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-home\") pod \"simulator-w4-f73e89-kserve-775bd847dc-qz5h2\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") " pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:58:53.564678 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.564643    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-dshm\") pod \"simulator-w4-f73e89-kserve-775bd847dc-qz5h2\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") " pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:58:53.564678 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.564668    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-tmp-dir\") pod \"simulator-w4-f73e89-kserve-775bd847dc-qz5h2\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") " pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:58:53.565195 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.565061    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-tmp-dir\") pod \"simulator-w4-f73e89-kserve-775bd847dc-qz5h2\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") " pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:58:53.565300 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.565274    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-model-cache\") pod \"simulator-w4-f73e89-kserve-775bd847dc-qz5h2\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") " pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:58:53.565530 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.565487    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-home\") pod \"simulator-w4-f73e89-kserve-775bd847dc-qz5h2\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") " pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:58:53.567842 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.567820    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-dshm\") pod \"simulator-w4-f73e89-kserve-775bd847dc-qz5h2\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") " pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:58:53.568121 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.568100    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-tls-certs\") pod \"simulator-w4-f73e89-kserve-775bd847dc-qz5h2\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") " pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:58:53.627601 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.627571    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grh6s\" (UniqueName: \"kubernetes.io/projected/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-kube-api-access-grh6s\") pod \"simulator-w4-f73e89-kserve-775bd847dc-qz5h2\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") " pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:58:53.700520 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:53.700447    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:58:54.077066 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:54.077021    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"]
Nov 01 05:58:54.079813 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:58:54.079787    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e8d332c_35b4_40b7_bf20_7623f1bc19d9.slice/crio-c9202bca7bf2a3ae65153a07000d5a512a5b350381b3f1ae8763f367cd25ba5f WatchSource:0}: Error finding container c9202bca7bf2a3ae65153a07000d5a512a5b350381b3f1ae8763f367cd25ba5f: Status 404 returned error can't find the container with id c9202bca7bf2a3ae65153a07000d5a512a5b350381b3f1ae8763f367cd25ba5f
Nov 01 05:58:54.317655 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:54.317616    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2" event={"ID":"8e8d332c-35b4-40b7-bf20-7623f1bc19d9","Type":"ContainerStarted","Data":"9f2a97c51a512caab00022ca8643390cc7557ed25aded6fd9c10a5ee5886777e"}
Nov 01 05:58:54.317805 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:54.317668    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2" event={"ID":"8e8d332c-35b4-40b7-bf20-7623f1bc19d9","Type":"ContainerStarted","Data":"c9202bca7bf2a3ae65153a07000d5a512a5b350381b3f1ae8763f367cd25ba5f"}
Nov 01 05:58:54.438334 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:54.438249    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2" podStartSLOduration=1.438230176 podStartE2EDuration="1.438230176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:54.437294575 +0000 UTC m=+771.648118084" watchObservedRunningTime="2026-11-01 05:58:54.438230176 +0000 UTC m=+771.649053687"
Nov 01 05:58:54.700588 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:54.700516    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:58:55.878458 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:58:55.878420    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:58:56.561728 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:56.561696    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:57.574867 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:57.574837    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:58.105621 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.105590    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"]
Nov 01 05:58:58.109232 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.109217    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.163206 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.163163    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0d2807-openshift-default-dockercfg-wkdwl\""
Nov 01 05:58:58.207859 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.207824    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:58.209929 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.209908    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-credential-socket\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.209999 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.209942    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k29ln\" (UniqueName: \"kubernetes.io/projected/680af3cd-644f-459b-81bf-686fdc62b198-kube-api-access-k29ln\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.210039 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.210016    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/680af3cd-644f-459b-81bf-686fdc62b198-istio-token\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.210078 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.210046    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-workload-socket\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.210078 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.210064    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/680af3cd-644f-459b-81bf-686fdc62b198-istio-podinfo\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.210149 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.210090    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-workload-certs\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.210228 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.210213    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-istio-data\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.210265 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.210237    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-istio-envoy\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.210308 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.210266    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/680af3cd-644f-459b-81bf-686fdc62b198-istiod-ca-cert\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.217966 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.217941    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"]
Nov 01 05:58:58.311865 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.311828    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-istio-data\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.311865 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.311866    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-istio-envoy\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.312124 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.311897    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/680af3cd-644f-459b-81bf-686fdc62b198-istiod-ca-cert\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.312124 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.311964    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-credential-socket\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.312124 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.311986    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k29ln\" (UniqueName: \"kubernetes.io/projected/680af3cd-644f-459b-81bf-686fdc62b198-kube-api-access-k29ln\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.312124 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.312035    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/680af3cd-644f-459b-81bf-686fdc62b198-istio-token\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.312124 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.312082    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-workload-socket\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.312124 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.312106    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/680af3cd-644f-459b-81bf-686fdc62b198-istio-podinfo\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.312496 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.312237    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-workload-certs\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.312496 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.312370    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-istio-data\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.312496 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.312439    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-credential-socket\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.312726 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.312703    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-workload-socket\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.312844 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.312701    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-workload-certs\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.312844 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.312748    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/680af3cd-644f-459b-81bf-686fdc62b198-istiod-ca-cert\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.314207 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.314183    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-istio-envoy\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.314596 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.314574    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/680af3cd-644f-459b-81bf-686fdc62b198-istio-podinfo\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.373747 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.373677    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/680af3cd-644f-459b-81bf-686fdc62b198-istio-token\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.374426 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.374401    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k29ln\" (UniqueName: \"kubernetes.io/projected/680af3cd-644f-459b-81bf-686fdc62b198-kube-api-access-k29ln\") pod \"e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") " pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.422252 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.422225    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:58.761151 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.761058    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:58:58.858922 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.858901    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"]
Nov 01 05:58:58.861880 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:58:58.861852    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680af3cd_644f_459b_81bf_686fdc62b198.slice/crio-34e31a808679d69ceb5cc0a32d9b60f3d99a27e5c8e1fefe293d1d37d2fca6dd WatchSource:0}: Error finding container 34e31a808679d69ceb5cc0a32d9b60f3d99a27e5c8e1fefe293d1d37d2fca6dd: Status 404 returned error can't find the container with id 34e31a808679d69ceb5cc0a32d9b60f3d99a27e5c8e1fefe293d1d37d2fca6dd
Nov 01 05:58:58.864281 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.864248    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:58:58.864373 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.864312    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:58:58.864373 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:58.864367    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:58:59.353477 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:59.353444    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k" event={"ID":"680af3cd-644f-459b-81bf-686fdc62b198","Type":"ContainerStarted","Data":"57f27d5a1a6076a06b7ec3c7a9a746cd1b66533d09968679d6c85920a711be38"}
Nov 01 05:58:59.353646 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:59.353497    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k" event={"ID":"680af3cd-644f-459b-81bf-686fdc62b198","Type":"ContainerStarted","Data":"34e31a808679d69ceb5cc0a32d9b60f3d99a27e5c8e1fefe293d1d37d2fca6dd"}
Nov 01 05:58:59.422809 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:59.422775    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:59.427525 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:59.427504    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:58:59.476176 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:58:59.476124    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k" podStartSLOduration=1.476110631 podStartE2EDuration="1.476110631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:58:59.474409289 +0000 UTC m=+776.685232801" watchObservedRunningTime="2026-11-01 05:58:59.476110631 +0000 UTC m=+776.686934139"
Nov 01 05:59:00.359689 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:00.359653    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:59:00.360762 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:00.360742    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:59:00.380751 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:00.380688    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:01.038195 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:01.038162    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:01.272886 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:01.272847    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:01.700664 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:01.700626    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:02.306783 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:02.306744    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:02.474566 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:02.474479    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:02.634685 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:02.634636    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:03.700931 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:03.700892    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:59:03.713281 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:03.713258    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:59:04.389541 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:04.389512    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 05:59:04.905233 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:04.905184    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:05.503157 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:05.503120    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:05.676577 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:05.676533    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:06.067169 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:06.067134    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns"]
Nov 01 05:59:06.071843 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:06.071823    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns"
Nov 01 05:59:06.127395 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:06.127342    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0d2807\""
Nov 01 05:59:06.177637 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:06.177609    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0d2807-dockercfg-rpr7q\""
Nov 01 05:59:06.179436 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:06.179414    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns"]
Nov 01 05:59:06.206928 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:06.206902    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b887a3-3119-4b1a-a642-9a5c7c21ff5a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns\" (UID: \"97b887a3-3119-4b1a-a642-9a5c7c21ff5a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns"
Nov 01 05:59:06.207055 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:06.206944    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkhfz\" (UniqueName: \"kubernetes.io/projected/97b887a3-3119-4b1a-a642-9a5c7c21ff5a-kube-api-access-lkhfz\") pod \"payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns\" (UID: \"97b887a3-3119-4b1a-a642-9a5c7c21ff5a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns"
Nov 01 05:59:06.307709 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:06.307675    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b887a3-3119-4b1a-a642-9a5c7c21ff5a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns\" (UID: \"97b887a3-3119-4b1a-a642-9a5c7c21ff5a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns"
Nov 01 05:59:06.307709 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:06.307721    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkhfz\" (UniqueName: \"kubernetes.io/projected/97b887a3-3119-4b1a-a642-9a5c7c21ff5a-kube-api-access-lkhfz\") pod \"payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns\" (UID: \"97b887a3-3119-4b1a-a642-9a5c7c21ff5a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns"
Nov 01 05:59:06.308469 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:06.308450    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b887a3-3119-4b1a-a642-9a5c7c21ff5a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns\" (UID: \"97b887a3-3119-4b1a-a642-9a5c7c21ff5a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns"
Nov 01 05:59:06.369909 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:06.369880    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkhfz\" (UniqueName: \"kubernetes.io/projected/97b887a3-3119-4b1a-a642-9a5c7c21ff5a-kube-api-access-lkhfz\") pod \"payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns\" (UID: \"97b887a3-3119-4b1a-a642-9a5c7c21ff5a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns"
Nov 01 05:59:06.385119 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:06.385095    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns"
Nov 01 05:59:06.757917 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:06.757894    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns"]
Nov 01 05:59:06.760163 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:59:06.760132    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b887a3_3119_4b1a_a642_9a5c7c21ff5a.slice/crio-fa61a027ea09182966bb6a061e2e924fdf80af8346eefd1776731f25839790cb WatchSource:0}: Error finding container fa61a027ea09182966bb6a061e2e924fdf80af8346eefd1776731f25839790cb: Status 404 returned error can't find the container with id fa61a027ea09182966bb6a061e2e924fdf80af8346eefd1776731f25839790cb
Nov 01 05:59:06.963929 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:06.963897    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:07.397831 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:07.397785    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns" event={"ID":"97b887a3-3119-4b1a-a642-9a5c7c21ff5a","Type":"ContainerStarted","Data":"febabc79ba6c36db7ac1d2c0bea7045c9f9c8ed0be52353c2d278bd9c950ac99"}
Nov 01 05:59:07.398310 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:07.397869    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns" event={"ID":"97b887a3-3119-4b1a-a642-9a5c7c21ff5a","Type":"ContainerStarted","Data":"fa61a027ea09182966bb6a061e2e924fdf80af8346eefd1776731f25839790cb"}
Nov 01 05:59:07.398310 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:07.397891    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns"
Nov 01 05:59:07.516775 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:07.516718    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns" podStartSLOduration=1.5167020340000001 podStartE2EDuration="1.516702034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:59:07.514868646 +0000 UTC m=+784.725692172" watchObservedRunningTime="2026-11-01 05:59:07.516702034 +0000 UTC m=+784.727525542"
Nov 01 05:59:09.039229 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.039190    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"]
Nov 01 05:59:09.043124 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.043100    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.132883 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.132842    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/074ba060-2a84-479d-9d8b-92987fb010e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.133166 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.133135    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.133283 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.133182    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/074ba060-2a84-479d-9d8b-92987fb010e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.133283 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.133227    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.133283 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.133273    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.133432 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.133315    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/074ba060-2a84-479d-9d8b-92987fb010e9-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.133432 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.133361    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.133508 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.133436    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzrqq\" (UniqueName: \"kubernetes.io/projected/074ba060-2a84-479d-9d8b-92987fb010e9-kube-api-access-gzrqq\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.133508 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.133478    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.234609 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.234578    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzrqq\" (UniqueName: \"kubernetes.io/projected/074ba060-2a84-479d-9d8b-92987fb010e9-kube-api-access-gzrqq\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.234778 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.234622    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.234778 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.234685    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/074ba060-2a84-479d-9d8b-92987fb010e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.234778 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.234716    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.234778 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.234739    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/074ba060-2a84-479d-9d8b-92987fb010e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.234988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.234781    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.234988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.234824    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.234988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.234861    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/074ba060-2a84-479d-9d8b-92987fb010e9-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.234988 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.234892    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.235187 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.235164    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.235246 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.235202    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.235297 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.235269    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.235434 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.235409    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.235628 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.235603    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/074ba060-2a84-479d-9d8b-92987fb010e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.237166 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.237146    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.237350 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.237311    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/074ba060-2a84-479d-9d8b-92987fb010e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.250354 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.250301    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"]
Nov 01 05:59:09.296188 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.296124    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/074ba060-2a84-479d-9d8b-92987fb010e9-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.296381 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.296355    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzrqq\" (UniqueName: \"kubernetes.io/projected/074ba060-2a84-479d-9d8b-92987fb010e9-kube-api-access-gzrqq\") pod \"maas-default-gateway-openshift-default-855f9574cf-qbzfs\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.356574 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.356542    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:09.785985 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.785958    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"]
Nov 01 05:59:09.787911 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:59:09.787884    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod074ba060_2a84_479d_9d8b_92987fb010e9.slice/crio-8edb94752d02bbb836b19933af0d96f5291390ef4158ca1a0806b2495e513f8b WatchSource:0}: Error finding container 8edb94752d02bbb836b19933af0d96f5291390ef4158ca1a0806b2495e513f8b: Status 404 returned error can't find the container with id 8edb94752d02bbb836b19933af0d96f5291390ef4158ca1a0806b2495e513f8b
Nov 01 05:59:09.790190 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.790157    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:59:09.790267 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.790227    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:59:09.790267 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:09.790259    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:59:10.416960 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:10.416922    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs" event={"ID":"074ba060-2a84-479d-9d8b-92987fb010e9","Type":"ContainerStarted","Data":"bb33628316f779c952e16a48ffe64795df08196c9b221cad5055fcc016b56ec7"}
Nov 01 05:59:10.417431 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:10.416978    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs" event={"ID":"074ba060-2a84-479d-9d8b-92987fb010e9","Type":"ContainerStarted","Data":"8edb94752d02bbb836b19933af0d96f5291390ef4158ca1a0806b2495e513f8b"}
Nov 01 05:59:10.646034 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:10.645979    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs" podStartSLOduration=2.6459651280000003 podStartE2EDuration="2.645965128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:59:10.594645274 +0000 UTC m=+787.805468810" watchObservedRunningTime="2026-11-01 05:59:10.645965128 +0000 UTC m=+787.856788637"
Nov 01 05:59:11.362345 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:11.357367    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:11.362558 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:11.362366    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qbzfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 01 05:59:11.362558 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:11.362436    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs" podUID="074ba060-2a84-479d-9d8b-92987fb010e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 01 05:59:11.804753 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:11.804656    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:12.365749 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:12.365709    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qbzfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 01 05:59:12.366036 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:12.366008    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs" podUID="074ba060-2a84-479d-9d8b-92987fb010e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 01 05:59:13.362774 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:13.362733    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qbzfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 01 05:59:13.368432 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:13.368391    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs" podUID="074ba060-2a84-479d-9d8b-92987fb010e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 01 05:59:14.369489 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:14.369456    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qbzfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 01 05:59:14.370063 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:14.370033    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs" podUID="074ba060-2a84-479d-9d8b-92987fb010e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 01 05:59:15.368817 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:15.368775    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qbzfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 01 05:59:15.369026 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:15.368836    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs" podUID="074ba060-2a84-479d-9d8b-92987fb010e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 01 05:59:15.661402 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:15.660016    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:16.357266 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:16.357227    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qbzfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 01 05:59:16.357437 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:16.357296    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs" podUID="074ba060-2a84-479d-9d8b-92987fb010e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 01 05:59:16.569504 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:16.569470    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:18.179082 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:18.179046    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:18.179682 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:18.179489    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:18.180354 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:18.180309    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 05:59:18.406387 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:18.406355    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns"
Nov 01 05:59:21.023378 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:21.023345    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"]
Nov 01 05:59:21.027156 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:21.027139    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"
Nov 01 05:59:21.078284 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:21.078258    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a8793d-01dc-42a1-83bd-37f944265ce1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg\" (UID: \"f5a8793d-01dc-42a1-83bd-37f944265ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"
Nov 01 05:59:21.078435 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:21.078306    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5a8793d-01dc-42a1-83bd-37f944265ce1-maas-api-tls\") pod \"maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg\" (UID: \"f5a8793d-01dc-42a1-83bd-37f944265ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"
Nov 01 05:59:21.078435 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:21.078420    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5a8793d-01dc-42a1-83bd-37f944265ce1-metrics-certs\") pod \"maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg\" (UID: \"f5a8793d-01dc-42a1-83bd-37f944265ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"
Nov 01 05:59:21.078525 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:21.078482    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlwz9\" (UniqueName: \"kubernetes.io/projected/f5a8793d-01dc-42a1-83bd-37f944265ce1-kube-api-access-hlwz9\") pod \"maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg\" (UID: \"f5a8793d-01dc-42a1-83bd-37f944265ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"
Nov 01 05:59:21.080454 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:21.080437    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0d2807-serving-cert\""
Nov 01 05:59:21.133264 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:21.133239    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"]
Nov 01 05:59:21.179913 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:21.179881    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a8793d-01dc-42a1-83bd-37f944265ce1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg\" (UID: \"f5a8793d-01dc-42a1-83bd-37f944265ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"
Nov 01 05:59:21.180055 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:21.179952    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5a8793d-01dc-42a1-83bd-37f944265ce1-maas-api-tls\") pod \"maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg\" (UID: \"f5a8793d-01dc-42a1-83bd-37f944265ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"
Nov 01 05:59:21.180055 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:21.179989    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5a8793d-01dc-42a1-83bd-37f944265ce1-metrics-certs\") pod \"maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg\" (UID: \"f5a8793d-01dc-42a1-83bd-37f944265ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"
Nov 01 05:59:21.180169 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:21.180066    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlwz9\" (UniqueName: \"kubernetes.io/projected/f5a8793d-01dc-42a1-83bd-37f944265ce1-kube-api-access-hlwz9\") pod \"maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg\" (UID: \"f5a8793d-01dc-42a1-83bd-37f944265ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"
Nov 01 05:59:21.180169 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:59:21.180095    5619 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-serving-cert: secret "maas-api-e2e-inf-a-0d2807-serving-cert" not found
Nov 01 05:59:21.180261 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:59:21.180182    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5a8793d-01dc-42a1-83bd-37f944265ce1-maas-api-tls podName:f5a8793d-01dc-42a1-83bd-37f944265ce1 nodeName:}" failed. No retries permitted until 2026-11-01 05:59:21.680159586 +0000 UTC m=+798.890983097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f5a8793d-01dc-42a1-83bd-37f944265ce1-maas-api-tls") pod "maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg" (UID: "f5a8793d-01dc-42a1-83bd-37f944265ce1") : secret "maas-api-e2e-inf-a-0d2807-serving-cert" not found
Nov 01 05:59:21.180654 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:21.180631    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a8793d-01dc-42a1-83bd-37f944265ce1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg\" (UID: \"f5a8793d-01dc-42a1-83bd-37f944265ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"
Nov 01 05:59:21.182564 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:21.182540    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5a8793d-01dc-42a1-83bd-37f944265ce1-metrics-certs\") pod \"maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg\" (UID: \"f5a8793d-01dc-42a1-83bd-37f944265ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"
Nov 01 05:59:21.219409 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:21.219379    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:21.242040 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:21.242011    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlwz9\" (UniqueName: \"kubernetes.io/projected/f5a8793d-01dc-42a1-83bd-37f944265ce1-kube-api-access-hlwz9\") pod \"maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg\" (UID: \"f5a8793d-01dc-42a1-83bd-37f944265ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"
Nov 01 05:59:21.377783 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:21.377751    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:21.528588 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:21.528541    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:21.685991 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:21.685909    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5a8793d-01dc-42a1-83bd-37f944265ce1-maas-api-tls\") pod \"maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg\" (UID: \"f5a8793d-01dc-42a1-83bd-37f944265ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"
Nov 01 05:59:21.688484 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:21.688456    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5a8793d-01dc-42a1-83bd-37f944265ce1-maas-api-tls\") pod \"maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg\" (UID: \"f5a8793d-01dc-42a1-83bd-37f944265ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"
Nov 01 05:59:21.940958 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:21.940881    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"
Nov 01 05:59:22.323466 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:22.323438    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"]
Nov 01 05:59:22.326343 ip-10-0-129-213 kubenswrapper[5619]: W1101 05:59:22.326295    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a8793d_01dc_42a1_83bd_37f944265ce1.slice/crio-ef84abec390eba129a7b3c3e43c73d64a9d7568e19d22d2e425d3f20a1fd3bb6 WatchSource:0}: Error finding container ef84abec390eba129a7b3c3e43c73d64a9d7568e19d22d2e425d3f20a1fd3bb6: Status 404 returned error can't find the container with id ef84abec390eba129a7b3c3e43c73d64a9d7568e19d22d2e425d3f20a1fd3bb6
Nov 01 05:59:22.483278 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:22.483235    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg" event={"ID":"f5a8793d-01dc-42a1-83bd-37f944265ce1","Type":"ContainerStarted","Data":"ef84abec390eba129a7b3c3e43c73d64a9d7568e19d22d2e425d3f20a1fd3bb6"}
Nov 01 05:59:23.489556 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:23.489464    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg" event={"ID":"f5a8793d-01dc-42a1-83bd-37f944265ce1","Type":"ContainerStarted","Data":"e6f77c99ca64e1d8cf87fac2cea11a33fca69c30a62b28426358e6c4f2adee91"}
Nov 01 05:59:23.489556 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:23.489535    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"
Nov 01 05:59:23.608710 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:23.608652    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg" podStartSLOduration=2.777312187 podStartE2EDuration="3.608638045s" totalImagesPullingTime="831.325858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 05:59:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 05:59:23.607966804 +0000 UTC m=+800.818790335" watchObservedRunningTime="2026-11-01 05:59:23.608638045 +0000 UTC m=+800.819461554"
Nov 01 05:59:24.178140 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:24.178107    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:25.881201 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:59:25.881165    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:59:26.172383 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:26.172293    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:29.501140 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:29.501115    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"
Nov 01 05:59:30.958156 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:30.958107    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:35.806850 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:35.806806    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:40.569287 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:40.569244    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:41.245183 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:41.245138    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:41.590907 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:41.590877    5619 generic.go:356] "Generic (PLEG): container finished" podID="680af3cd-644f-459b-81bf-686fdc62b198" containerID="57f27d5a1a6076a06b7ec3c7a9a746cd1b66533d09968679d6c85920a711be38" exitCode=0
Nov 01 05:59:41.591382 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:41.590952    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k" event={"ID":"680af3cd-644f-459b-81bf-686fdc62b198","Type":"ContainerDied","Data":"57f27d5a1a6076a06b7ec3c7a9a746cd1b66533d09968679d6c85920a711be38"}
Nov 01 05:59:41.591458 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:41.591437    5619 scope.go:122] "RemoveContainer" containerID="57f27d5a1a6076a06b7ec3c7a9a746cd1b66533d09968679d6c85920a711be38"
Nov 01 05:59:41.592752 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:41.592728    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:59:41.592841 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:41.592792    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:59:41.592841 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:41.592824    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 05:59:42.598365 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:42.598316    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k" event={"ID":"680af3cd-644f-459b-81bf-686fdc62b198","Type":"ContainerStarted","Data":"2fcfb1f692f66039c3596e1a41ca1775222c998663b4b8eabd19212a81215f09"}
Nov 01 05:59:43.422881 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:43.422853    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:59:43.423864 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:43.423840    5619 patch_prober.go:29] interesting pod/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 01 05:59:43.423940 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:43.423892    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k" podUID="680af3cd-644f-459b-81bf-686fdc62b198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 01 05:59:44.423037 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:44.423003    5619 patch_prober.go:29] interesting pod/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 01 05:59:44.423437 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:44.423051    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k" podUID="680af3cd-644f-459b-81bf-686fdc62b198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 01 05:59:44.906789 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:44.906745    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:45.363353 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:45.362636    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:59:45.395906 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:45.395875    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:45.422799 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:45.422771    5619 patch_prober.go:29] interesting pod/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 01 05:59:45.422945 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:45.422817    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k" podUID="680af3cd-644f-459b-81bf-686fdc62b198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 01 05:59:45.769405 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:45.764902    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:46.422889 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:46.422859    5619 patch_prober.go:29] interesting pod/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 01 05:59:46.423136 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:46.423114    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k" podUID="680af3cd-644f-459b-81bf-686fdc62b198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 01 05:59:47.423284 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:47.422807    5619 patch_prober.go:29] interesting pod/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 01 05:59:47.423284 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:47.422876    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k" podUID="680af3cd-644f-459b-81bf-686fdc62b198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 01 05:59:48.427370 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:48.427339    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:59:48.430769 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:48.430747    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 05:59:50.267791 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:50.267746    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:53.014174 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:53.014130    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:53.523734 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:53.523694    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:55.268394 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:55.268353    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 05:59:55.879542 ip-10-0-129-213 kubenswrapper[5619]: E1101 05:59:55.879501    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 05:59:57.081832 ip-10-0-129-213 kubenswrapper[5619]: I1101 05:59:57.081795    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:00.054946 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:00.054906    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:04.051618 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.051579    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"]
Nov 01 06:00:04.056140 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.056122    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.113396 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.113362    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-cda04f-openshift-default-dockercfg-sbp8q\""
Nov 01 06:00:04.146719 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.146684    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:04.171146 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.167931    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"]
Nov 01 06:00:04.190053 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.190021    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-workload-certs\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.190250 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.190086    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-credential-socket\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.190250 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.190197    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-podinfo\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.190250 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.190226    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istiod-ca-cert\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.190464 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.190270    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-data\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.190464 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.190371    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-workload-socket\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.190464 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.190407    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-token\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.190628 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.190520    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbm9k\" (UniqueName: \"kubernetes.io/projected/a1bfd841-06b0-48b4-9037-d6c77ddf0521-kube-api-access-nbm9k\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.190628 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.190572    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-envoy\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.291541 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.291508    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-workload-socket\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.291541 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.291542    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-token\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.291815 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.291580    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbm9k\" (UniqueName: \"kubernetes.io/projected/a1bfd841-06b0-48b4-9037-d6c77ddf0521-kube-api-access-nbm9k\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.291815 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.291604    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-envoy\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.291815 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.291645    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-workload-certs\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.291815 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.291688    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-credential-socket\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.291815 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.291765    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-podinfo\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.291815 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.291807    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istiod-ca-cert\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.292158 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.291979    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-data\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.292158 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.292104    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-workload-certs\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.292257 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.292208    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-credential-socket\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.292310 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.292263    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-workload-socket\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.292409 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.292369    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-data\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.292497 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.292477    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istiod-ca-cert\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.293992 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.293971    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-envoy\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.294313 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.294296    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-podinfo\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.355560 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.355529    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbm9k\" (UniqueName: \"kubernetes.io/projected/a1bfd841-06b0-48b4-9037-d6c77ddf0521-kube-api-access-nbm9k\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.358452 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.358422    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-token\") pod \"e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") " pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.369985 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.369951    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:04.408820 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.408789    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:04.806875 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.806849    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"]
Nov 01 06:00:04.809465 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:00:04.809437    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1bfd841_06b0_48b4_9037_d6c77ddf0521.slice/crio-30f5ea7673460f47f29508a1eb982b15914af8a58178def61f896c2c61e62cdf WatchSource:0}: Error finding container 30f5ea7673460f47f29508a1eb982b15914af8a58178def61f896c2c61e62cdf: Status 404 returned error can't find the container with id 30f5ea7673460f47f29508a1eb982b15914af8a58178def61f896c2c61e62cdf
Nov 01 06:00:04.814920 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.814887    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:00:04.815008 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.814956    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:00:04.815008 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.814984    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:00:04.860547 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:04.860517    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:05.718945 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:05.718899    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb" event={"ID":"a1bfd841-06b0-48b4-9037-d6c77ddf0521","Type":"ContainerStarted","Data":"71ae52201a399b1f9a8bdb6a54ea7c2467a605b14455cac50651193653f5012a"}
Nov 01 06:00:05.719487 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:05.719009    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb" event={"ID":"a1bfd841-06b0-48b4-9037-d6c77ddf0521","Type":"ContainerStarted","Data":"30f5ea7673460f47f29508a1eb982b15914af8a58178def61f896c2c61e62cdf"}
Nov 01 06:00:05.894386 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:05.894333    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb" podStartSLOduration=1.894306839 podStartE2EDuration="1.894306839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:05.892633893 +0000 UTC m=+843.103457401" watchObservedRunningTime="2026-11-01 06:00:05.894306839 +0000 UTC m=+843.105130347"
Nov 01 06:00:06.370127 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:06.370094    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:06.375758 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:06.375731    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:06.724872 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:06.724787    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:06.726041 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:06.726017    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:07.019481 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:07.019386    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:08.798453 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:08.798414    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:08.979964 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:08.979918    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:09.119709 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:09.119664    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:09.836174 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:09.836142    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:12.035086 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:12.035046    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:12.202341 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:12.202287    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:14.690662 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:14.690607    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:16.009859 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:16.009824    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:19.461946 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:19.461907    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:24.281350 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:24.281286    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:25.882509 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:00:25.882457    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:00:29.062514 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:29.062474    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:33.511558 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:33.511522    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"]
Nov 01 06:00:33.515630 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:33.515613    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"
Nov 01 06:00:33.568641 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:33.568609    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-cda04f-serving-cert\""
Nov 01 06:00:33.621559 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:33.621523    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"]
Nov 01 06:00:33.642638 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:33.642605    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfd6m\" (UniqueName: \"kubernetes.io/projected/04cd4917-baa4-421d-9638-d66d4b93ca79-kube-api-access-mfd6m\") pod \"maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7\" (UID: \"04cd4917-baa4-421d-9638-d66d4b93ca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"
Nov 01 06:00:33.642822 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:33.642668    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04cd4917-baa4-421d-9638-d66d4b93ca79-maas-api-tls\") pod \"maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7\" (UID: \"04cd4917-baa4-421d-9638-d66d4b93ca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"
Nov 01 06:00:33.642822 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:33.642794    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cd4917-baa4-421d-9638-d66d4b93ca79-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7\" (UID: \"04cd4917-baa4-421d-9638-d66d4b93ca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"
Nov 01 06:00:33.642938 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:33.642852    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04cd4917-baa4-421d-9638-d66d4b93ca79-metrics-certs\") pod \"maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7\" (UID: \"04cd4917-baa4-421d-9638-d66d4b93ca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"
Nov 01 06:00:33.707973 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:33.707936    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:33.744489 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:33.744447    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfd6m\" (UniqueName: \"kubernetes.io/projected/04cd4917-baa4-421d-9638-d66d4b93ca79-kube-api-access-mfd6m\") pod \"maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7\" (UID: \"04cd4917-baa4-421d-9638-d66d4b93ca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"
Nov 01 06:00:33.744698 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:33.744532    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04cd4917-baa4-421d-9638-d66d4b93ca79-maas-api-tls\") pod \"maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7\" (UID: \"04cd4917-baa4-421d-9638-d66d4b93ca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"
Nov 01 06:00:33.744698 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:33.744582    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cd4917-baa4-421d-9638-d66d4b93ca79-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7\" (UID: \"04cd4917-baa4-421d-9638-d66d4b93ca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"
Nov 01 06:00:33.744698 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:33.744623    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04cd4917-baa4-421d-9638-d66d4b93ca79-metrics-certs\") pod \"maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7\" (UID: \"04cd4917-baa4-421d-9638-d66d4b93ca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"
Nov 01 06:00:33.744698 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:00:33.744671    5619 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-serving-cert: secret "maas-api-e2e-inf-b-cda04f-serving-cert" not found
Nov 01 06:00:33.744917 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:00:33.744743    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04cd4917-baa4-421d-9638-d66d4b93ca79-maas-api-tls podName:04cd4917-baa4-421d-9638-d66d4b93ca79 nodeName:}" failed. No retries permitted until 2026-11-01 06:00:34.244727263 +0000 UTC m=+871.455550750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/04cd4917-baa4-421d-9638-d66d4b93ca79-maas-api-tls") pod "maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7" (UID: "04cd4917-baa4-421d-9638-d66d4b93ca79") : secret "maas-api-e2e-inf-b-cda04f-serving-cert" not found
Nov 01 06:00:33.745301 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:33.745276    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cd4917-baa4-421d-9638-d66d4b93ca79-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7\" (UID: \"04cd4917-baa4-421d-9638-d66d4b93ca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"
Nov 01 06:00:33.747134 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:33.747110    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04cd4917-baa4-421d-9638-d66d4b93ca79-metrics-certs\") pod \"maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7\" (UID: \"04cd4917-baa4-421d-9638-d66d4b93ca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"
Nov 01 06:00:33.808721 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:33.808644    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfd6m\" (UniqueName: \"kubernetes.io/projected/04cd4917-baa4-421d-9638-d66d4b93ca79-kube-api-access-mfd6m\") pod \"maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7\" (UID: \"04cd4917-baa4-421d-9638-d66d4b93ca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"
Nov 01 06:00:33.865657 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:33.865618    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:34.013827 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:34.013792    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:34.251825 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:34.251781    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04cd4917-baa4-421d-9638-d66d4b93ca79-maas-api-tls\") pod \"maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7\" (UID: \"04cd4917-baa4-421d-9638-d66d4b93ca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"
Nov 01 06:00:34.254214 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:34.254190    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04cd4917-baa4-421d-9638-d66d4b93ca79-maas-api-tls\") pod \"maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7\" (UID: \"04cd4917-baa4-421d-9638-d66d4b93ca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"
Nov 01 06:00:34.430159 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:34.430114    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"
Nov 01 06:00:34.811297 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:34.811272    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"]
Nov 01 06:00:34.813771 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:00:34.813739    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cd4917_baa4_421d_9638_d66d4b93ca79.slice/crio-3eb1b957f0739b69a483834853fbe510b26ac8b610060ba98701ad72df666619 WatchSource:0}: Error finding container 3eb1b957f0739b69a483834853fbe510b26ac8b610060ba98701ad72df666619: Status 404 returned error can't find the container with id 3eb1b957f0739b69a483834853fbe510b26ac8b610060ba98701ad72df666619
Nov 01 06:00:34.891012 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:34.890974    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7" event={"ID":"04cd4917-baa4-421d-9638-d66d4b93ca79","Type":"ContainerStarted","Data":"3eb1b957f0739b69a483834853fbe510b26ac8b610060ba98701ad72df666619"}
Nov 01 06:00:35.898481 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:35.898441    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7" event={"ID":"04cd4917-baa4-421d-9638-d66d4b93ca79","Type":"ContainerStarted","Data":"143c9f260c88167a8dd1febac339ca7daf98f23b4ec6feff13b8ea75c7ea7e94"}
Nov 01 06:00:35.898859 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:35.898585    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"
Nov 01 06:00:36.020184 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:36.020077    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7" podStartSLOduration=2.1327357989999998 podStartE2EDuration="3.020062625s" totalImagesPullingTime="887.326826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:00:36.018516793 +0000 UTC m=+873.229340303" watchObservedRunningTime="2026-11-01 06:00:36.020062625 +0000 UTC m=+873.230886133"
Nov 01 06:00:38.680487 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:38.680448    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:41.909952 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:41.909920    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"
Nov 01 06:00:43.608539 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:43.608504    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:48.381729 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:48.381686    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:50.176119 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:50.176081    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:50.322298 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:50.322247    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:50.859391 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:50.859354    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:50.999252 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:50.999174    5619 generic.go:356] "Generic (PLEG): container finished" podID="a1bfd841-06b0-48b4-9037-d6c77ddf0521" containerID="71ae52201a399b1f9a8bdb6a54ea7c2467a605b14455cac50651193653f5012a" exitCode=0
Nov 01 06:00:50.999252 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:50.999239    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb" event={"ID":"a1bfd841-06b0-48b4-9037-d6c77ddf0521","Type":"ContainerDied","Data":"71ae52201a399b1f9a8bdb6a54ea7c2467a605b14455cac50651193653f5012a"}
Nov 01 06:00:50.999607 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:50.999591    5619 scope.go:122] "RemoveContainer" containerID="71ae52201a399b1f9a8bdb6a54ea7c2467a605b14455cac50651193653f5012a"
Nov 01 06:00:51.000883 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:51.000851    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:00:51.000968 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:51.000926    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:00:51.000968 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:51.000958    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:00:51.725164 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:51.725134    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:52.007140 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:52.007056    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb" event={"ID":"a1bfd841-06b0-48b4-9037-d6c77ddf0521","Type":"ContainerStarted","Data":"ac28211bf0c9792bfa38fb5dabb7175c47be643202213beff2f76f9dce24c6f2"}
Nov 01 06:00:52.370758 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:52.370722    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:52.376009 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:52.375514    5619 patch_prober.go:29] interesting pod/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 01 06:00:52.376009 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:52.375598    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb" podUID="a1bfd841-06b0-48b4-9037-d6c77ddf0521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 01 06:00:53.205895 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:53.205848    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:00:53.374345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:53.373687    5619 patch_prober.go:29] interesting pod/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 01 06:00:53.374345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:53.373767    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb" podUID="a1bfd841-06b0-48b4-9037-d6c77ddf0521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 01 06:00:54.374714 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:54.374680    5619 patch_prober.go:29] interesting pod/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 01 06:00:54.375285 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:54.375253    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb" podUID="a1bfd841-06b0-48b4-9037-d6c77ddf0521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 01 06:00:55.373215 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:55.373181    5619 patch_prober.go:29] interesting pod/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 01 06:00:55.373499 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:55.373471    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb" podUID="a1bfd841-06b0-48b4-9037-d6c77ddf0521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 01 06:00:55.906497 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:00:55.906457    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:00:56.371745 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:56.371709    5619 patch_prober.go:29] interesting pod/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 01 06:00:56.372022 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:56.371996    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb" podUID="a1bfd841-06b0-48b4-9037-d6c77ddf0521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 01 06:00:57.375194 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:57.375163    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:57.375690 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:57.375602    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:57.376357 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:57.376318    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:00:58.075023 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:00:58.074989    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:02.852462 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:02.852423    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:07.588853 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:07.588810    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:09.765110 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:09.765073    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:12.609452 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:12.609360    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:16.346415 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:16.346379    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:17.350085 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:17.350051    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:22.161275 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:22.161215    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:25.879345 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:01:25.879295    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:01:26.875931 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:26.875898    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:28.564821 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.564781    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"]
Nov 01 06:01:28.569192 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.569169    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:28.625183 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.625137    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0d2807\"/\"openshift-service-ca.crt\""
Nov 01 06:01:28.648739 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.648710    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0d2807\"/\"kube-root-ca.crt\""
Nov 01 06:01:28.675921 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.675890    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0d2807\"/\"default-dockercfg-9dknk\""
Nov 01 06:01:28.676053 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.676009    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0d2807\"/\"test-model-0d2807-kserve-self-signed-certs\""
Nov 01 06:01:28.678110 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.678092    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"]
Nov 01 06:01:28.679704 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.679678    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:28.698549 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.698521    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc76ed5-591e-4230-8c24-8c199d46d87e-tls-certs\") pod \"test-model-0d2807-kserve-69b99c649-22st6\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") " pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:28.698669 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.698612    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-model-cache\") pod \"test-model-0d2807-kserve-69b99c649-22st6\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") " pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:28.698669 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.698652    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-home\") pod \"test-model-0d2807-kserve-69b99c649-22st6\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") " pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:28.698766 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.698698    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-dshm\") pod \"test-model-0d2807-kserve-69b99c649-22st6\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") " pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:28.698805 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.698793    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrvmn\" (UniqueName: \"kubernetes.io/projected/0bc76ed5-591e-4230-8c24-8c199d46d87e-kube-api-access-hrvmn\") pod \"test-model-0d2807-kserve-69b99c649-22st6\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") " pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:28.698840 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.698827    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-tmp-dir\") pod \"test-model-0d2807-kserve-69b99c649-22st6\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") " pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:28.800444 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.800401    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-tmp-dir\") pod \"test-model-0d2807-kserve-69b99c649-22st6\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") " pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:28.800638 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.800544    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc76ed5-591e-4230-8c24-8c199d46d87e-tls-certs\") pod \"test-model-0d2807-kserve-69b99c649-22st6\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") " pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:28.800638 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.800575    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-model-cache\") pod \"test-model-0d2807-kserve-69b99c649-22st6\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") " pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:28.800638 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.800602    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-home\") pod \"test-model-0d2807-kserve-69b99c649-22st6\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") " pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:28.800810 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.800646    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-dshm\") pod \"test-model-0d2807-kserve-69b99c649-22st6\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") " pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:28.800810 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.800723    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrvmn\" (UniqueName: \"kubernetes.io/projected/0bc76ed5-591e-4230-8c24-8c199d46d87e-kube-api-access-hrvmn\") pod \"test-model-0d2807-kserve-69b99c649-22st6\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") " pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:28.800925 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.800902    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-tmp-dir\") pod \"test-model-0d2807-kserve-69b99c649-22st6\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") " pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:28.800963 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.800915    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-home\") pod \"test-model-0d2807-kserve-69b99c649-22st6\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") " pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:28.800997 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.800972    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-model-cache\") pod \"test-model-0d2807-kserve-69b99c649-22st6\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") " pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:28.802953 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.802926    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-dshm\") pod \"test-model-0d2807-kserve-69b99c649-22st6\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") " pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:28.803118 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.803101    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc76ed5-591e-4230-8c24-8c199d46d87e-tls-certs\") pod \"test-model-0d2807-kserve-69b99c649-22st6\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") " pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:28.862295 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.862262    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrvmn\" (UniqueName: \"kubernetes.io/projected/0bc76ed5-591e-4230-8c24-8c199d46d87e-kube-api-access-hrvmn\") pod \"test-model-0d2807-kserve-69b99c649-22st6\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") " pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:28.882106 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:28.882085    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:29.266343 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:29.266295    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"]
Nov 01 06:01:29.267778 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:01:29.267752    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bc76ed5_591e_4230_8c24_8c199d46d87e.slice/crio-890b97d12f5eae0de7246fe7e212a23f4e91001a96f14f571286f320b38cda59 WatchSource:0}: Error finding container 890b97d12f5eae0de7246fe7e212a23f4e91001a96f14f571286f320b38cda59: Status 404 returned error can't find the container with id 890b97d12f5eae0de7246fe7e212a23f4e91001a96f14f571286f320b38cda59
Nov 01 06:01:30.213914 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:30.213877    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6" event={"ID":"0bc76ed5-591e-4230-8c24-8c199d46d87e","Type":"ContainerStarted","Data":"b56e63713bbc2c7881fb7a6fb60e85eb83a3de5ac584398b65f4eafb635ac030"}
Nov 01 06:01:30.214350 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:30.213922    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6" event={"ID":"0bc76ed5-591e-4230-8c24-8c199d46d87e","Type":"ContainerStarted","Data":"890b97d12f5eae0de7246fe7e212a23f4e91001a96f14f571286f320b38cda59"}
Nov 01 06:01:30.332827 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:30.332778    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6" podStartSLOduration=2.332764752 podStartE2EDuration="2.332764752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:01:30.331897223 +0000 UTC m=+927.542720731" watchObservedRunningTime="2026-11-01 06:01:30.332764752 +0000 UTC m=+927.543588262"
Nov 01 06:01:30.883041 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:30.883005    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:31.668896 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:31.668861    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:36.531438 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:36.531402    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:38.882849 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:38.882818    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:38.895551 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:38.895524    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:39.271771 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:39.271690    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:01:41.253229 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:41.253192    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:43.211004 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:43.210969    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:43.401653 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:43.401616    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:45.961843 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:45.961801    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:47.541658 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:47.541620    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:49.423824 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:49.423788    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:50.046954 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:50.046919    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:50.294168 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:50.294130    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:50.803640 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:50.803602    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:53.636017 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.635975    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"]
Nov 01 06:01:53.639718 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.639702    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:01:53.723348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.723291    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:53.743866 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.743842    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f73e89\"/\"distinct-2-w4-f73e89-kserve-self-signed-certs\""
Nov 01 06:01:53.746101 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.746073    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"]
Nov 01 06:01:53.780508 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.780469    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-tls-certs\") pod \"distinct-2-w4-f73e89-kserve-bd6475978-nkqm8\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:01:53.780696 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.780541    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-dshm\") pod \"distinct-2-w4-f73e89-kserve-bd6475978-nkqm8\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:01:53.780696 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.780648    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9zbk\" (UniqueName: \"kubernetes.io/projected/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-kube-api-access-d9zbk\") pod \"distinct-2-w4-f73e89-kserve-bd6475978-nkqm8\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:01:53.780839 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.780693    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-home\") pod \"distinct-2-w4-f73e89-kserve-bd6475978-nkqm8\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:01:53.780839 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.780742    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-model-cache\") pod \"distinct-2-w4-f73e89-kserve-bd6475978-nkqm8\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:01:53.780951 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.780938    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-tmp-dir\") pod \"distinct-2-w4-f73e89-kserve-bd6475978-nkqm8\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:01:53.881883 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.881846    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-dshm\") pod \"distinct-2-w4-f73e89-kserve-bd6475978-nkqm8\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:01:53.882048 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.881892    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9zbk\" (UniqueName: \"kubernetes.io/projected/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-kube-api-access-d9zbk\") pod \"distinct-2-w4-f73e89-kserve-bd6475978-nkqm8\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:01:53.882048 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.882006    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-home\") pod \"distinct-2-w4-f73e89-kserve-bd6475978-nkqm8\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:01:53.882048 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.882043    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-model-cache\") pod \"distinct-2-w4-f73e89-kserve-bd6475978-nkqm8\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:01:53.882247 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.882222    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-tmp-dir\") pod \"distinct-2-w4-f73e89-kserve-bd6475978-nkqm8\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:01:53.882304 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.882271    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-tls-certs\") pod \"distinct-2-w4-f73e89-kserve-bd6475978-nkqm8\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:01:53.882505 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.882483    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-home\") pod \"distinct-2-w4-f73e89-kserve-bd6475978-nkqm8\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:01:53.882576 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.882509    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-model-cache\") pod \"distinct-2-w4-f73e89-kserve-bd6475978-nkqm8\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:01:53.882576 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.882556    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-tmp-dir\") pod \"distinct-2-w4-f73e89-kserve-bd6475978-nkqm8\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:01:53.884210 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.884189    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-dshm\") pod \"distinct-2-w4-f73e89-kserve-bd6475978-nkqm8\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:01:53.884755 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.884736    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-tls-certs\") pod \"distinct-2-w4-f73e89-kserve-bd6475978-nkqm8\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:01:53.943891 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.943821    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9zbk\" (UniqueName: \"kubernetes.io/projected/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-kube-api-access-d9zbk\") pod \"distinct-2-w4-f73e89-kserve-bd6475978-nkqm8\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") " pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:01:53.952065 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:53.952041    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:01:54.041583 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:54.041476    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:54.210817 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:54.210730    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:54.331364 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:54.331304    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"]
Nov 01 06:01:54.338553 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:01:54.338526    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb255dfb_b714_4f2e_98c2_e8aa8695b9da.slice/crio-84b0d6f7de2ba89430125dc9ea7c1d02bdbaeefa1aa77447a1b87920241b7c42 WatchSource:0}: Error finding container 84b0d6f7de2ba89430125dc9ea7c1d02bdbaeefa1aa77447a1b87920241b7c42: Status 404 returned error can't find the container with id 84b0d6f7de2ba89430125dc9ea7c1d02bdbaeefa1aa77447a1b87920241b7c42
Nov 01 06:01:54.357183 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:54.357151    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:55.347502 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:55.347471    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8" event={"ID":"fb255dfb-b714-4f2e-98c2-e8aa8695b9da","Type":"ContainerStarted","Data":"e7e55af8de06f327898327f3140311064a0136561ccb971644c0231dbe87b1da"}
Nov 01 06:01:55.347921 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:55.347511    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8" event={"ID":"fb255dfb-b714-4f2e-98c2-e8aa8695b9da","Type":"ContainerStarted","Data":"84b0d6f7de2ba89430125dc9ea7c1d02bdbaeefa1aa77447a1b87920241b7c42"}
Nov 01 06:01:55.473451 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:55.473391    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8" podStartSLOduration=2.473373768 podStartE2EDuration="2.473373768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:01:55.471712302 +0000 UTC m=+952.682535810" watchObservedRunningTime="2026-11-01 06:01:55.473373768 +0000 UTC m=+952.684197279"
Nov 01 06:01:55.878359 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:55.877432    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:55.882501 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:01:55.882468    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:01:55.952480 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:55.952451    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:01:57.290415 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:57.290380    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:01:57.468090 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:01:57.468049    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:00.892745 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:00.892703    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:03.952369 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:03.952336    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:02:03.965422 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:03.965397    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:02:04.406317 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:04.406284    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:02:05.661859 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:05.661820    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:10.462176 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:10.462136    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:15.316902 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:15.316861    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:16.758145 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:16.758102    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:17.874971 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:17.874930    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"]
Nov 01 06:02:17.879643 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:17.879621    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:17.938865 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:17.938835    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"]
Nov 01 06:02:17.944784 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:17.944756    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:17.944961 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:17.944790    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:17.944961 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:17.944827    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23561314-57a9-4def-a052-32c23ccdd1f0-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:17.945095 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:17.944985    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56spz\" (UniqueName: \"kubernetes.io/projected/23561314-57a9-4def-a052-32c23ccdd1f0-kube-api-access-56spz\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:17.945095 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:17.945009    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23561314-57a9-4def-a052-32c23ccdd1f0-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:17.945095 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:17.945049    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:17.945271 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:17.945114    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23561314-57a9-4def-a052-32c23ccdd1f0-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:17.945271 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:17.945139    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:17.945271 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:17.945160    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:17.986549 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:17.986511    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-14a6e4-openshift-default-dockercfg-vxhc4\""
Nov 01 06:02:18.046754 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.046721    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56spz\" (UniqueName: \"kubernetes.io/projected/23561314-57a9-4def-a052-32c23ccdd1f0-kube-api-access-56spz\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:18.046922 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.046761    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23561314-57a9-4def-a052-32c23ccdd1f0-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:18.046922 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.046815    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:18.046922 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.046855    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23561314-57a9-4def-a052-32c23ccdd1f0-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:18.046922 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.046894    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:18.047126 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.046924    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:18.047126 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.047003    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:18.047126 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.047032    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:18.047126 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.047086    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23561314-57a9-4def-a052-32c23ccdd1f0-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:18.047373 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.047275    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:18.047373 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.047342    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:18.047463 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.047416    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:18.047463 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.047444    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:18.047669 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.047648    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23561314-57a9-4def-a052-32c23ccdd1f0-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:18.049439 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.049418    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:18.049439 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.049428    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23561314-57a9-4def-a052-32c23ccdd1f0-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:18.110785 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.110755    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23561314-57a9-4def-a052-32c23ccdd1f0-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:18.110931 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.110766    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56spz\" (UniqueName: \"kubernetes.io/projected/23561314-57a9-4def-a052-32c23ccdd1f0-kube-api-access-56spz\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:18.193746 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.193653    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:18.624161 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.624131    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"]
Nov 01 06:02:18.626164 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:02:18.626126    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23561314_57a9_4def_a052_32c23ccdd1f0.slice/crio-b9609a754fa664849bfaf9f2f0d073a4ee50f092ca8bc16acbd6016108b494bd WatchSource:0}: Error finding container b9609a754fa664849bfaf9f2f0d073a4ee50f092ca8bc16acbd6016108b494bd: Status 404 returned error can't find the container with id b9609a754fa664849bfaf9f2f0d073a4ee50f092ca8bc16acbd6016108b494bd
Nov 01 06:02:18.628509 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.628481    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:02:18.628575 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.628542    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:02:18.628575 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:18.628572    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:02:19.488410 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:19.488367    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk" event={"ID":"23561314-57a9-4def-a052-32c23ccdd1f0","Type":"ContainerStarted","Data":"6c345b575f576b09ada6941038a28c2158ba8b70c4778ec7312bdd361c10b233"}
Nov 01 06:02:19.488800 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:19.488442    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk" event={"ID":"23561314-57a9-4def-a052-32c23ccdd1f0","Type":"ContainerStarted","Data":"b9609a754fa664849bfaf9f2f0d073a4ee50f092ca8bc16acbd6016108b494bd"}
Nov 01 06:02:19.672213 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:19.672135    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk" podStartSLOduration=2.672112983 podStartE2EDuration="2.672112983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:02:19.670013979 +0000 UTC m=+976.880837491" watchObservedRunningTime="2026-11-01 06:02:19.672112983 +0000 UTC m=+976.882936495"
Nov 01 06:02:20.193880 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:20.193843    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:20.195535 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:20.195503    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 01 06:02:20.195666 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:20.195565    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk" podUID="23561314-57a9-4def-a052-32c23ccdd1f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 01 06:02:20.302522 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:20.302480    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:21.194604 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:21.194558    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 01 06:02:21.195119 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:21.194617    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk" podUID="23561314-57a9-4def-a052-32c23ccdd1f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 01 06:02:22.194824 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:22.194779    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 01 06:02:22.195248 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:22.194848    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk" podUID="23561314-57a9-4def-a052-32c23ccdd1f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 01 06:02:22.990990 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:22.990954    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"]
Nov 01 06:02:22.994860 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:22.994842    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.048153 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.048123    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9c8fc5-openshift-default-dockercfg-6mn84\""
Nov 01 06:02:23.101573 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.101538    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"]
Nov 01 06:02:23.120030 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.119994    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.120194 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.120044    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.120194 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.120065    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwvk9\" (UniqueName: \"kubernetes.io/projected/e4c10775-05ae-437a-bb34-422cefafc913-kube-api-access-hwvk9\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.120194 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.120086    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.120194 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.120111    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c10775-05ae-437a-bb34-422cefafc913-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.120194 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.120134    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.120194 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.120164    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.120194 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.120193    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c10775-05ae-437a-bb34-422cefafc913-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.120436 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.120237    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c10775-05ae-437a-bb34-422cefafc913-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.194608 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.194571    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 01 06:02:23.194781 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.194639    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk" podUID="23561314-57a9-4def-a052-32c23ccdd1f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 01 06:02:23.221403 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.221363    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.221789 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.221418    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.221789 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.221442    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwvk9\" (UniqueName: \"kubernetes.io/projected/e4c10775-05ae-437a-bb34-422cefafc913-kube-api-access-hwvk9\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.221789 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.221469    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.221789 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.221510    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c10775-05ae-437a-bb34-422cefafc913-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.221789 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.221548    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.221789 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.221599    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.221789 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.221649    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c10775-05ae-437a-bb34-422cefafc913-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.221789 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.221726    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c10775-05ae-437a-bb34-422cefafc913-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.222058 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.221918    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.222613 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.222370    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.222613 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.222475    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.222613 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.222489    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c10775-05ae-437a-bb34-422cefafc913-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.222613 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.222548    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.223865 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.223839    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.224565 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.224542    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c10775-05ae-437a-bb34-422cefafc913-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.285790 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.285708    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c10775-05ae-437a-bb34-422cefafc913-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.285959 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.285938    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwvk9\" (UniqueName: \"kubernetes.io/projected/e4c10775-05ae-437a-bb34-422cefafc913-kube-api-access-hwvk9\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.309906 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.309877    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:23.777566 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.777525    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"]
Nov 01 06:02:23.782424 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:02:23.782393    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c10775_05ae_437a_bb34_422cefafc913.slice/crio-ad5665ceaee04466599af5d45326f9a3dc2c97ee5538edc7d0f03ea98359dc47 WatchSource:0}: Error finding container ad5665ceaee04466599af5d45326f9a3dc2c97ee5538edc7d0f03ea98359dc47: Status 404 returned error can't find the container with id ad5665ceaee04466599af5d45326f9a3dc2c97ee5538edc7d0f03ea98359dc47
Nov 01 06:02:23.784904 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.784863    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:02:23.784991 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.784955    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:02:23.785034 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:23.785001    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:02:24.194575 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:24.194536    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 01 06:02:24.194765 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:24.194599    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk" podUID="23561314-57a9-4def-a052-32c23ccdd1f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 01 06:02:24.523407 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:24.523297    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj" event={"ID":"e4c10775-05ae-437a-bb34-422cefafc913","Type":"ContainerStarted","Data":"d8a1230f940aec93b2736675e9e8d2288597478a29430e824c91e41b24f23e15"}
Nov 01 06:02:24.523825 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:24.523806    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj" event={"ID":"e4c10775-05ae-437a-bb34-422cefafc913","Type":"ContainerStarted","Data":"ad5665ceaee04466599af5d45326f9a3dc2c97ee5538edc7d0f03ea98359dc47"}
Nov 01 06:02:24.681312 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:24.681235    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj" podStartSLOduration=2.6812134480000003 podStartE2EDuration="2.681213448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:02:24.680625181 +0000 UTC m=+981.891448691" watchObservedRunningTime="2026-11-01 06:02:24.681213448 +0000 UTC m=+981.892036951"
Nov 01 06:02:25.071349 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:25.061952    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:25.224364 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:25.224306    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:25.310500 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:25.310452    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:25.312111 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:25.312082    5619 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 01 06:02:25.312250 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:25.312135    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj" podUID="e4c10775-05ae-437a-bb34-422cefafc913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 01 06:02:25.724976 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:25.724938    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:25.899491 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:02:25.899452    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:02:26.235480 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:26.235449    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:02:26.235672 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:26.235515    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk" podUID="23561314-57a9-4def-a052-32c23ccdd1f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:02:26.310925 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:26.310885    5619 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 01 06:02:26.311102 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:26.310963    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj" podUID="e4c10775-05ae-437a-bb34-422cefafc913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 01 06:02:26.445949 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:26.445913    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:26.446779 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:26.446753    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:26.447828 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:26.447803    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:02:27.310306 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:27.310270    5619 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 01 06:02:27.310733 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:27.310361    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj" podUID="e4c10775-05ae-437a-bb34-422cefafc913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 01 06:02:28.310565 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:28.310532    5619 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 01 06:02:28.311031 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:28.310584    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj" podUID="e4c10775-05ae-437a-bb34-422cefafc913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 01 06:02:29.310864 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:29.310831    5619 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 01 06:02:29.311280 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:29.310886    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj" podUID="e4c10775-05ae-437a-bb34-422cefafc913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 01 06:02:30.104063 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:30.104017    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:30.568218 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:30.567988    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:30.568844 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:30.568821    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:30.569979 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:30.569955    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:02:34.864666 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:34.864615    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:39.660144 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:39.660099    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:44.446352 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:44.446243    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:49.174919 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:49.174887    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:54.104870 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:54.104829    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:54.720704 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:54.720664    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:02:55.880781 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:02:55.880747    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:02:58.876697 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:02:58.876657    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:03.668954 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:03.668920    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:05.065148 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:05.065112    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:06.280770 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:06.280733    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:06.528742 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:06.528709    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:06.877142 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:06.877105    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:07.185938 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:07.185848    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:07.346634 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:07.346597    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:07.497141 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:07.497046    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:08.680279 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:08.680243    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:11.152131 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:11.152079    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:11.320951 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:11.320913    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:11.472619 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:11.472534    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:11.781203 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:11.781120    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:12.004217 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:12.004186    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:12.155081 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:12.155050    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:13.995677 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:13.995639    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:14.362495 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:14.362457    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:14.558544 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:14.558512    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:17.059643 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:17.059595    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:17.264795 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:17.264754    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:19.209121 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:19.209083    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:19.402638 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:19.402592    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:19.622945 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:19.622911    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s"]
Nov 01 06:03:19.626644 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:19.626627    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s"
Nov 01 06:03:19.735221 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:19.735169    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-58d523\""
Nov 01 06:03:19.735467 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:19.735445    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-58d523-dockercfg-6h5sw\""
Nov 01 06:03:19.776343 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:19.772672    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz82d\" (UniqueName: \"kubernetes.io/projected/6a5b6319-91ea-449a-8196-0738360b4ec6-kube-api-access-hz82d\") pod \"payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s\" (UID: \"6a5b6319-91ea-449a-8196-0738360b4ec6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s"
Nov 01 06:03:19.776343 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:19.772760    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5b6319-91ea-449a-8196-0738360b4ec6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s\" (UID: \"6a5b6319-91ea-449a-8196-0738360b4ec6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s"
Nov 01 06:03:19.776656 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:19.776634    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s"]
Nov 01 06:03:19.874492 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:19.874388    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz82d\" (UniqueName: \"kubernetes.io/projected/6a5b6319-91ea-449a-8196-0738360b4ec6-kube-api-access-hz82d\") pod \"payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s\" (UID: \"6a5b6319-91ea-449a-8196-0738360b4ec6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s"
Nov 01 06:03:19.874492 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:19.874467    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5b6319-91ea-449a-8196-0738360b4ec6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s\" (UID: \"6a5b6319-91ea-449a-8196-0738360b4ec6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s"
Nov 01 06:03:19.875589 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:19.875543    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5b6319-91ea-449a-8196-0738360b4ec6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s\" (UID: \"6a5b6319-91ea-449a-8196-0738360b4ec6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s"
Nov 01 06:03:19.936843 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:19.936819    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz82d\" (UniqueName: \"kubernetes.io/projected/6a5b6319-91ea-449a-8196-0738360b4ec6-kube-api-access-hz82d\") pod \"payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s\" (UID: \"6a5b6319-91ea-449a-8196-0738360b4ec6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s"
Nov 01 06:03:19.939692 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:19.939666    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s"
Nov 01 06:03:20.319340 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:20.319298    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s"]
Nov 01 06:03:20.321403 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:03:20.321375    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5b6319_91ea_449a_8196_0738360b4ec6.slice/crio-224c8f8b824c149c733a57f832baf39338d5cb6177204c43ba0bf41110387e9d WatchSource:0}: Error finding container 224c8f8b824c149c733a57f832baf39338d5cb6177204c43ba0bf41110387e9d: Status 404 returned error can't find the container with id 224c8f8b824c149c733a57f832baf39338d5cb6177204c43ba0bf41110387e9d
Nov 01 06:03:20.894098 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:20.894055    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s" event={"ID":"6a5b6319-91ea-449a-8196-0738360b4ec6","Type":"ContainerStarted","Data":"58cf9949daa82df6c6d3f1cd8ce48d3c8779a5ff6e1936bd515cb5eee600669c"}
Nov 01 06:03:20.894359 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:20.894172    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s"
Nov 01 06:03:20.894359 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:20.894207    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s" event={"ID":"6a5b6319-91ea-449a-8196-0738360b4ec6","Type":"ContainerStarted","Data":"224c8f8b824c149c733a57f832baf39338d5cb6177204c43ba0bf41110387e9d"}
Nov 01 06:03:21.014297 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:21.014238    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s" podStartSLOduration=2.014223971 podStartE2EDuration="2.014223971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:03:21.012078389 +0000 UTC m=+1038.222901899" watchObservedRunningTime="2026-11-01 06:03:21.014223971 +0000 UTC m=+1038.225047481"
Nov 01 06:03:23.452933 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.452887    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"]
Nov 01 06:03:23.456992 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.456970    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.514664 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.514634    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.514664 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.514669    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.514874 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.514687    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.514874 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.514756    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d487e22-0f51-434a-98db-93e7a80b44ad-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.514958 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.514892    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.514958 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.514931    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.515043 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.514969    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqcgp\" (UniqueName: \"kubernetes.io/projected/5d487e22-0f51-434a-98db-93e7a80b44ad-kube-api-access-zqcgp\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.515043 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.514997    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.515153 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.515141    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.566447 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.566412    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"]
Nov 01 06:03:23.615805 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.615771    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.616011 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.615835    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.616011 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.615858    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.616011 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.615875    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.616011 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.615923    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d487e22-0f51-434a-98db-93e7a80b44ad-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.616011 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.615968    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.616294 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.616002    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.616294 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.616070    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqcgp\" (UniqueName: \"kubernetes.io/projected/5d487e22-0f51-434a-98db-93e7a80b44ad-kube-api-access-zqcgp\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.616294 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.616118    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.616294 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.616269    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.616477 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.616310    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.616477 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.616393    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.616759 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.616728    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.616882 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.616770    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d487e22-0f51-434a-98db-93e7a80b44ad-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.618373 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.618346    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.618606 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.618587    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.678094 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.678065    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqcgp\" (UniqueName: \"kubernetes.io/projected/5d487e22-0f51-434a-98db-93e7a80b44ad-kube-api-access-zqcgp\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.678199 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.678145    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:23.771106 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:23.771045    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:24.170040 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:24.169999    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:24.203662 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:24.203638    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"]
Nov 01 06:03:24.205038 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:03:24.205015    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d487e22_0f51_434a_98db_93e7a80b44ad.slice/crio-35a68f4013ad42df40a269f7e7954e7cbfd17d6d7cdadfb06a99ae8dff86f68d WatchSource:0}: Error finding container 35a68f4013ad42df40a269f7e7954e7cbfd17d6d7cdadfb06a99ae8dff86f68d: Status 404 returned error can't find the container with id 35a68f4013ad42df40a269f7e7954e7cbfd17d6d7cdadfb06a99ae8dff86f68d
Nov 01 06:03:24.208488 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:24.207992    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:03:24.208488 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:24.208058    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:03:24.208488 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:24.208094    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:03:24.918167 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:24.918128    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq" event={"ID":"5d487e22-0f51-434a-98db-93e7a80b44ad","Type":"ContainerStarted","Data":"93291367d64fd20e7c4a1d82a032ac409a1be9824edb6c31329fca3206a770f6"}
Nov 01 06:03:24.918582 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:24.918219    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq" event={"ID":"5d487e22-0f51-434a-98db-93e7a80b44ad","Type":"ContainerStarted","Data":"35a68f4013ad42df40a269f7e7954e7cbfd17d6d7cdadfb06a99ae8dff86f68d"}
Nov 01 06:03:25.094747 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:25.094693    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq" podStartSLOduration=2.094677537 podStartE2EDuration="2.094677537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:03:25.092383865 +0000 UTC m=+1042.303207394" watchObservedRunningTime="2026-11-01 06:03:25.094677537 +0000 UTC m=+1042.305501045"
Nov 01 06:03:25.773299 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:25.773265    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:25.776488 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:25.776459    5619 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 01 06:03:25.776722 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:25.776683    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq" podUID="5d487e22-0f51-434a-98db-93e7a80b44ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 01 06:03:25.899915 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:03:25.899855    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:03:26.774345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:26.773709    5619 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 01 06:03:26.774345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:26.773788    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq" podUID="5d487e22-0f51-434a-98db-93e7a80b44ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 01 06:03:27.775345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:27.774673    5619 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 01 06:03:27.775345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:27.774728    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq" podUID="5d487e22-0f51-434a-98db-93e7a80b44ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 01 06:03:28.774348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:28.771760    5619 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 01 06:03:28.774348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:28.771837    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq" podUID="5d487e22-0f51-434a-98db-93e7a80b44ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 01 06:03:29.112422 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:29.112382    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:29.772344 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:29.771723    5619 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 01 06:03:29.772344 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:29.771791    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq" podUID="5d487e22-0f51-434a-98db-93e7a80b44ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 01 06:03:30.891257 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:30.891216    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:30.892499 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:30.892476    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:30.893084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:30.893067    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:03:31.901878 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:31.901845    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s"
Nov 01 06:03:33.995823 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:33.995786    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:38.777113 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:38.777063    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:43.567875 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:43.567832    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:48.090431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:48.090394    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"]
Nov 01 06:03:48.094201 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:48.094179    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"
Nov 01 06:03:48.104406 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:48.104382    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"]
Nov 01 06:03:48.148157 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:48.148128    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-3986bf-serving-cert\""
Nov 01 06:03:48.160769 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:48.160740    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x45pj\" (UniqueName: \"kubernetes.io/projected/4bbaa266-0be4-4403-bbb2-8d4b552899c0-kube-api-access-x45pj\") pod \"maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg\" (UID: \"4bbaa266-0be4-4403-bbb2-8d4b552899c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"
Nov 01 06:03:48.160919 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:48.160796    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbaa266-0be4-4403-bbb2-8d4b552899c0-metrics-certs\") pod \"maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg\" (UID: \"4bbaa266-0be4-4403-bbb2-8d4b552899c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"
Nov 01 06:03:48.161010 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:48.160992    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bbaa266-0be4-4403-bbb2-8d4b552899c0-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg\" (UID: \"4bbaa266-0be4-4403-bbb2-8d4b552899c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"
Nov 01 06:03:48.161227 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:48.161199    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bbaa266-0be4-4403-bbb2-8d4b552899c0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg\" (UID: \"4bbaa266-0be4-4403-bbb2-8d4b552899c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"
Nov 01 06:03:48.214267 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:48.214236    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:48.262507 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:48.262470    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x45pj\" (UniqueName: \"kubernetes.io/projected/4bbaa266-0be4-4403-bbb2-8d4b552899c0-kube-api-access-x45pj\") pod \"maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg\" (UID: \"4bbaa266-0be4-4403-bbb2-8d4b552899c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"
Nov 01 06:03:48.262717 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:48.262536    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbaa266-0be4-4403-bbb2-8d4b552899c0-metrics-certs\") pod \"maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg\" (UID: \"4bbaa266-0be4-4403-bbb2-8d4b552899c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"
Nov 01 06:03:48.263589 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:48.263557    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bbaa266-0be4-4403-bbb2-8d4b552899c0-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg\" (UID: \"4bbaa266-0be4-4403-bbb2-8d4b552899c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"
Nov 01 06:03:48.263697 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:48.263636    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bbaa266-0be4-4403-bbb2-8d4b552899c0-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg\" (UID: \"4bbaa266-0be4-4403-bbb2-8d4b552899c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"
Nov 01 06:03:48.263769 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:48.263732    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bbaa266-0be4-4403-bbb2-8d4b552899c0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg\" (UID: \"4bbaa266-0be4-4403-bbb2-8d4b552899c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"
Nov 01 06:03:48.265677 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:48.265657    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbaa266-0be4-4403-bbb2-8d4b552899c0-metrics-certs\") pod \"maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg\" (UID: \"4bbaa266-0be4-4403-bbb2-8d4b552899c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"
Nov 01 06:03:48.266400 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:48.266382    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bbaa266-0be4-4403-bbb2-8d4b552899c0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg\" (UID: \"4bbaa266-0be4-4403-bbb2-8d4b552899c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"
Nov 01 06:03:48.324762 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:48.324733    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x45pj\" (UniqueName: \"kubernetes.io/projected/4bbaa266-0be4-4403-bbb2-8d4b552899c0-kube-api-access-x45pj\") pod \"maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg\" (UID: \"4bbaa266-0be4-4403-bbb2-8d4b552899c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"
Nov 01 06:03:48.362393 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:48.362303    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:48.410991 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:48.410965    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"
Nov 01 06:03:48.523236 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:48.521988    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:48.798177 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:48.798147    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"]
Nov 01 06:03:48.800089 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:03:48.800062    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bbaa266_0be4_4403_bbb2_8d4b552899c0.slice/crio-e784625c3e5801b121535f984c2f010902bf0706241c106ad39309751318ecd2 WatchSource:0}: Error finding container e784625c3e5801b121535f984c2f010902bf0706241c106ad39309751318ecd2: Status 404 returned error can't find the container with id e784625c3e5801b121535f984c2f010902bf0706241c106ad39309751318ecd2
Nov 01 06:03:48.801311 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:48.801293    5619 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 06:03:49.114365 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:49.114313    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg" event={"ID":"4bbaa266-0be4-4403-bbb2-8d4b552899c0","Type":"ContainerStarted","Data":"e784625c3e5801b121535f984c2f010902bf0706241c106ad39309751318ecd2"}
Nov 01 06:03:50.121596 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:50.121556    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg" event={"ID":"4bbaa266-0be4-4403-bbb2-8d4b552899c0","Type":"ContainerStarted","Data":"0bfc866e41a016e3574977ac0a1cdf1f7439db5d646e7745080d34ebbb531853"}
Nov 01 06:03:50.121998 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:50.121653    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"
Nov 01 06:03:50.241399 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:50.241313    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg" podStartSLOduration=2.383092822 podStartE2EDuration="3.241297755s" totalImagesPullingTime="858.204933ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:03:50.240725596 +0000 UTC m=+1067.451549108" watchObservedRunningTime="2026-11-01 06:03:50.241297755 +0000 UTC m=+1067.452121263"
Nov 01 06:03:53.116410 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:53.116362    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:53.322838 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:53.322801    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:53.473252 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:53.473160    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:53.967882 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:53.967850    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"]
Nov 01 06:03:53.971717 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:53.971697    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.031572 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.031545    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39efefc9-9afa-4664-91f3-a86554a6e761-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.031719 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.031576    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.031719 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.031596    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwdjq\" (UniqueName: \"kubernetes.io/projected/39efefc9-9afa-4664-91f3-a86554a6e761-kube-api-access-cwdjq\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.031836 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.031764    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39efefc9-9afa-4664-91f3-a86554a6e761-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.031836 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.031819    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39efefc9-9afa-4664-91f3-a86554a6e761-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.031920 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.031879    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.031984 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.031962    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.032070 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.032053    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.032130 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.032101    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.084372 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.082632    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"]
Nov 01 06:03:54.133307 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.133274    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39efefc9-9afa-4664-91f3-a86554a6e761-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.133745 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.133314    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39efefc9-9afa-4664-91f3-a86554a6e761-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.133745 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.133358    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.133745 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.133402    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.133745 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.133489    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.133745 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.133532    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.133745 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.133595    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39efefc9-9afa-4664-91f3-a86554a6e761-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.133745 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.133623    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.133745 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.133652    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwdjq\" (UniqueName: \"kubernetes.io/projected/39efefc9-9afa-4664-91f3-a86554a6e761-kube-api-access-cwdjq\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.134127 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.133912    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.134186 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.134107    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.134186 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.134137    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.134354 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.134314    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.134499 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.134473    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39efefc9-9afa-4664-91f3-a86554a6e761-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.135649 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.135624    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.135906 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.135888    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39efefc9-9afa-4664-91f3-a86554a6e761-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.197507 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.197478    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39efefc9-9afa-4664-91f3-a86554a6e761-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.198524 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.198507    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwdjq\" (UniqueName: \"kubernetes.io/projected/39efefc9-9afa-4664-91f3-a86554a6e761-kube-api-access-cwdjq\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.284798 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.284725    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:54.731505 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.731474    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"]
Nov 01 06:03:54.732829 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:03:54.732803    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39efefc9_9afa_4664_91f3_a86554a6e761.slice/crio-fd5c479dfc931d29c9a79f58d99d3a73449df2c6a222241cd70e4556ffb23693 WatchSource:0}: Error finding container fd5c479dfc931d29c9a79f58d99d3a73449df2c6a222241cd70e4556ffb23693: Status 404 returned error can't find the container with id fd5c479dfc931d29c9a79f58d99d3a73449df2c6a222241cd70e4556ffb23693
Nov 01 06:03:54.735533 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.735492    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:03:54.735681 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.735574    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:03:54.735681 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:54.735614    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:03:55.154223 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:55.154183    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5" event={"ID":"39efefc9-9afa-4664-91f3-a86554a6e761","Type":"ContainerStarted","Data":"90032855d51c0ebf15285bc95e0d3c4c0df337b05d80d6e1aea976bdf900a221"}
Nov 01 06:03:55.154635 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:55.154255    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5" event={"ID":"39efefc9-9afa-4664-91f3-a86554a6e761","Type":"ContainerStarted","Data":"fd5c479dfc931d29c9a79f58d99d3a73449df2c6a222241cd70e4556ffb23693"}
Nov 01 06:03:55.285372 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:55.285317    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:03:55.286581 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:55.286554    5619 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 01 06:03:55.286702 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:55.286613    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5" podUID="39efefc9-9afa-4664-91f3-a86554a6e761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 01 06:03:55.913954 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:03:55.913919    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:03:56.133216 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:56.133190    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"
Nov 01 06:03:56.276397 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:56.271433    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5" podStartSLOduration=3.271413876 podStartE2EDuration="3.271413876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:03:55.282159458 +0000 UTC m=+1072.492982967" watchObservedRunningTime="2026-11-01 06:03:56.271413876 +0000 UTC m=+1073.482237386"
Nov 01 06:03:56.285825 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:56.285235    5619 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 01 06:03:56.285825 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:56.285290    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5" podUID="39efefc9-9afa-4664-91f3-a86554a6e761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 01 06:03:57.285134 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:57.285092    5619 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 01 06:03:57.285640 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:57.285153    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5" podUID="39efefc9-9afa-4664-91f3-a86554a6e761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 01 06:03:58.209671 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:58.209635    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:03:58.291803 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:58.291765    5619 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 01 06:03:58.292337 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:58.291827    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5" podUID="39efefc9-9afa-4664-91f3-a86554a6e761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 01 06:03:59.285691 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:59.285651    5619 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 01 06:03:59.285882 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:59.285721    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5" podUID="39efefc9-9afa-4664-91f3-a86554a6e761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 01 06:03:59.789700 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:03:59.789648    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:00.286057 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:00.286019    5619 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 01 06:04:00.286246 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:00.286084    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5" podUID="39efefc9-9afa-4664-91f3-a86554a6e761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 01 06:04:01.285590 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:01.285553    5619 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 01 06:04:01.286074 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:01.285607    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5" podUID="39efefc9-9afa-4664-91f3-a86554a6e761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 01 06:04:02.289875 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:02.289838    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:04:02.290276 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:02.290116    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:04:02.290811 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:02.290792    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:04:03.017973 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.017940    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:03.697569 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.697536    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"]
Nov 01 06:04:03.702906 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.702882    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.810064 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.810030    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"]
Nov 01 06:04:03.816859 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.816827    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c53fe0e-52d4-48ca-b508-96aa30adea55-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.816994 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.816953    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.817059 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.817041    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.817264 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.817244    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.817302 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.817274    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.817302 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.817295    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.817417 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.817367    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.817417 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.817410    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbhdr\" (UniqueName: \"kubernetes.io/projected/6c53fe0e-52d4-48ca-b508-96aa30adea55-kube-api-access-hbhdr\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.817494 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.817466    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.918594 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.918544    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.918801 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.918598    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.918801 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.918629    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.918801 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.918671    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.918801 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.918703    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbhdr\" (UniqueName: \"kubernetes.io/projected/6c53fe0e-52d4-48ca-b508-96aa30adea55-kube-api-access-hbhdr\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.918801 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.918750    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.918801 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.918787    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c53fe0e-52d4-48ca-b508-96aa30adea55-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.919134 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.918850    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.919134 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.918945    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.919134 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.919110    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.919277 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.919130    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.919784 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.919753    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.920119 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.920096    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.920302 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.920273    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c53fe0e-52d4-48ca-b508-96aa30adea55-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.921859 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.921836    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.922382 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.922354    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.981087 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.981015    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:03.981659 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:03.981631    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbhdr\" (UniqueName: \"kubernetes.io/projected/6c53fe0e-52d4-48ca-b508-96aa30adea55-kube-api-access-hbhdr\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:04.019555 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:04.019226    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:04.485574 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:04:04.485353    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c53fe0e_52d4_48ca_b508_96aa30adea55.slice/crio-abccc7c22ee0f54a41f76ec204804ef68ec0498a13a53d73b6a56874bc350ac4 WatchSource:0}: Error finding container abccc7c22ee0f54a41f76ec204804ef68ec0498a13a53d73b6a56874bc350ac4: Status 404 returned error can't find the container with id abccc7c22ee0f54a41f76ec204804ef68ec0498a13a53d73b6a56874bc350ac4
Nov 01 06:04:04.493136 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:04.492395    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:04:04.493136 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:04.492484    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:04:04.493136 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:04.492525    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:04:04.502338 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:04.501944    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"]
Nov 01 06:04:05.232638 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:05.232600    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl" event={"ID":"6c53fe0e-52d4-48ca-b508-96aa30adea55","Type":"ContainerStarted","Data":"95eaa0df8c1e7738e25338365ebd98c8a2e1a4a680bf1dd564add80e4d53fc21"}
Nov 01 06:04:05.233031 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:05.232660    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl" event={"ID":"6c53fe0e-52d4-48ca-b508-96aa30adea55","Type":"ContainerStarted","Data":"abccc7c22ee0f54a41f76ec204804ef68ec0498a13a53d73b6a56874bc350ac4"}
Nov 01 06:04:05.374919 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:05.374846    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl" podStartSLOduration=2.374826563 podStartE2EDuration="2.374826563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:04:05.372721782 +0000 UTC m=+1082.583545292" watchObservedRunningTime="2026-11-01 06:04:05.374826563 +0000 UTC m=+1082.585650073"
Nov 01 06:04:06.019403 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:06.019369    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:06.021014 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:06.020988    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 01 06:04:06.021133 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:06.021070    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl" podUID="6c53fe0e-52d4-48ca-b508-96aa30adea55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 01 06:04:07.020285 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:07.020254    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 01 06:04:07.020664 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:07.020311    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl" podUID="6c53fe0e-52d4-48ca-b508-96aa30adea55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 01 06:04:07.816467 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:07.813819    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:08.019635 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:08.019596    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 01 06:04:08.019800 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:08.019662    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl" podUID="6c53fe0e-52d4-48ca-b508-96aa30adea55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 01 06:04:09.019947 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:09.019910    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 01 06:04:09.020438 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:09.019968    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl" podUID="6c53fe0e-52d4-48ca-b508-96aa30adea55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 01 06:04:10.020389 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:10.020356    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 01 06:04:10.020751 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:10.020412    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl" podUID="6c53fe0e-52d4-48ca-b508-96aa30adea55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 01 06:04:11.169807 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:11.169724    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:11.170768 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:11.170742    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:11.171434 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:11.171415    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:04:12.666955 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:12.666915    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:17.436624 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:17.436586    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:17.629911 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:17.629877    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:22.163930 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:22.163885    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:24.021079 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:24.021038    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"]
Nov 01 06:04:24.021509 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:24.021258    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2" podUID="2670ba1b-cdaa-48f4-9808-b34fa5632cbd" containerName="istio-proxy" containerID="cri-o://a082f75ccc48904e6c02e06c26a00f14db4c5918ee6874e26a2c97e2dcb43f08" gracePeriod=30
Nov 01 06:04:25.943156 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:04:25.942987    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:04:26.972982 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:26.972942    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:29.315595 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.315563    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 06:04:29.374469 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.374438    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-podinfo\") pod \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") "
Nov 01 06:04:29.374638 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.374482    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-envoy\") pod \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") "
Nov 01 06:04:29.374638 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.374523    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-workload-socket\") pod \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") "
Nov 01 06:04:29.374734 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.374641    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-data\") pod \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") "
Nov 01 06:04:29.374734 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.374700    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-workload-certs\") pod \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") "
Nov 01 06:04:29.374829 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.374774    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-workload-socket" pod "2670ba1b-cdaa-48f4-9808-b34fa5632cbd" (UID: "2670ba1b-cdaa-48f4-9808-b34fa5632cbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:29.374890 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.374869    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-kube-api-access-v8ndb\" (UniqueName: \"kubernetes.io/projected/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-kube-api-access-v8ndb\") pod \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") "
Nov 01 06:04:29.374950 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.374886    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-data" pod "2670ba1b-cdaa-48f4-9808-b34fa5632cbd" (UID: "2670ba1b-cdaa-48f4-9808-b34fa5632cbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:29.374950 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.374918    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-token\" (UniqueName: \"kubernetes.io/projected/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-token\") pod \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") "
Nov 01 06:04:29.375056 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.374953    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istiod-ca-cert\") pod \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") "
Nov 01 06:04:29.375056 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.374985    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-workload-certs" pod "2670ba1b-cdaa-48f4-9808-b34fa5632cbd" (UID: "2670ba1b-cdaa-48f4-9808-b34fa5632cbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:29.375056 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.374997    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-credential-socket\") pod \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\" (UID: \"2670ba1b-cdaa-48f4-9808-b34fa5632cbd\") "
Nov 01 06:04:29.375243 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.375200    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-credential-socket" pod "2670ba1b-cdaa-48f4-9808-b34fa5632cbd" (UID: "2670ba1b-cdaa-48f4-9808-b34fa5632cbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:29.375421 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.375383    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istiod-ca-cert" pod "2670ba1b-cdaa-48f4-9808-b34fa5632cbd" (UID: "2670ba1b-cdaa-48f4-9808-b34fa5632cbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:04:29.375566 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.375538    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:29.375566 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.375556    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:29.375566 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.375565    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:29.375750 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.375575    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:29.375750 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.375588    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:29.376908 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.376880    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-envoy" pod "2670ba1b-cdaa-48f4-9808-b34fa5632cbd" (UID: "2670ba1b-cdaa-48f4-9808-b34fa5632cbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:04:29.377089 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.377070    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-token" pod "2670ba1b-cdaa-48f4-9808-b34fa5632cbd" (UID: "2670ba1b-cdaa-48f4-9808-b34fa5632cbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:04:29.377145 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.377128    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-podinfo" pod "2670ba1b-cdaa-48f4-9808-b34fa5632cbd" (UID: "2670ba1b-cdaa-48f4-9808-b34fa5632cbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:04:29.377248 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.377228    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-kube-api-access-v8ndb" pod "2670ba1b-cdaa-48f4-9808-b34fa5632cbd" (UID: "2670ba1b-cdaa-48f4-9808-b34fa5632cbd"). InnerVolumeSpecName "kube-api-access-v8ndb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:04:29.408295 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.408262    5619 generic.go:356] "Generic (PLEG): container finished" podID="2670ba1b-cdaa-48f4-9808-b34fa5632cbd" containerID="a082f75ccc48904e6c02e06c26a00f14db4c5918ee6874e26a2c97e2dcb43f08" exitCode=0
Nov 01 06:04:29.408446 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.408310    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"
Nov 01 06:04:29.408446 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.408351    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2" event={"ID":"2670ba1b-cdaa-48f4-9808-b34fa5632cbd","Type":"ContainerDied","Data":"a082f75ccc48904e6c02e06c26a00f14db4c5918ee6874e26a2c97e2dcb43f08"}
Nov 01 06:04:29.408446 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.408410    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2" event={"ID":"2670ba1b-cdaa-48f4-9808-b34fa5632cbd","Type":"ContainerDied","Data":"4a07e82085e8938bc628d5ceef06e6545ec8e0e5095010ae61eead89158e84ee"}
Nov 01 06:04:29.408446 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.408443    5619 scope.go:122] "RemoveContainer" containerID="a082f75ccc48904e6c02e06c26a00f14db4c5918ee6874e26a2c97e2dcb43f08"
Nov 01 06:04:29.419187 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.419163    5619 scope.go:122] "RemoveContainer" containerID="a082f75ccc48904e6c02e06c26a00f14db4c5918ee6874e26a2c97e2dcb43f08"
Nov 01 06:04:29.419490 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:04:29.419471    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a082f75ccc48904e6c02e06c26a00f14db4c5918ee6874e26a2c97e2dcb43f08\": container with ID starting with a082f75ccc48904e6c02e06c26a00f14db4c5918ee6874e26a2c97e2dcb43f08 not found: ID does not exist" containerID="a082f75ccc48904e6c02e06c26a00f14db4c5918ee6874e26a2c97e2dcb43f08"
Nov 01 06:04:29.419552 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.419502    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a082f75ccc48904e6c02e06c26a00f14db4c5918ee6874e26a2c97e2dcb43f08"} err="failed to get container status \"a082f75ccc48904e6c02e06c26a00f14db4c5918ee6874e26a2c97e2dcb43f08\": rpc error: code = NotFound desc = could not find container \"a082f75ccc48904e6c02e06c26a00f14db4c5918ee6874e26a2c97e2dcb43f08\": container with ID starting with a082f75ccc48904e6c02e06c26a00f14db4c5918ee6874e26a2c97e2dcb43f08 not found: ID does not exist"
Nov 01 06:04:29.476359 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.476314    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:29.476359 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.476358    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:29.476567 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.476374    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8ndb\" (UniqueName: \"kubernetes.io/projected/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-kube-api-access-v8ndb\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:29.476567 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.476387    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2670ba1b-cdaa-48f4-9808-b34fa5632cbd-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:29.606962 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.606929    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"]
Nov 01 06:04:29.612221 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.612193    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-c8mt2"]
Nov 01 06:04:29.720562 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:29.720527    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:30.669005 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:30.668967    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:30.963156 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:30.963067    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2670ba1b-cdaa-48f4-9808-b34fa5632cbd" path="/var/lib/kubelet/pods/2670ba1b-cdaa-48f4-9808-b34fa5632cbd/volumes"
Nov 01 06:04:31.756961 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:31.756923    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:36.578806 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:36.578763    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:37.867590 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:37.867550    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:39.149652 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:39.149600    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:39.299045 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:39.299007    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:39.469576 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:39.469479    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:39.791776 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:39.791685    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:39.949615 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:39.949579    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:40.110344 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:40.110070    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:41.198416 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:41.198385    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:41.347033 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:41.346999    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:41.801436 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:41.801402    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:43.430700 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:43.430668    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:44.306096 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:44.306060    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:44.870484 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:44.870445    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:45.020403 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:45.020357    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:45.176430 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:45.176308    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:46.301301 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:46.301269    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s"]
Nov 01 06:04:46.557889 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:46.557814    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:46.750646 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:46.750617    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:46.901009 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:46.900977    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:50.420504 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:50.420466    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:50.614151 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:50.614108    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:50.775402 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:50.766394    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:51.301883 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:51.301830    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s" podUID="6a5b6319-91ea-449a-8196-0738360b4ec6" containerName="payload-pre-processing" containerID="cri-o://58cf9949daa82df6c6d3f1cd8ce48d3c8779a5ff6e1936bd515cb5eee600669c" gracePeriod=25
Nov 01 06:04:51.551645 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:51.551608    5619 generic.go:356] "Generic (PLEG): container finished" podID="6a5b6319-91ea-449a-8196-0738360b4ec6" containerID="58cf9949daa82df6c6d3f1cd8ce48d3c8779a5ff6e1936bd515cb5eee600669c" exitCode=0
Nov 01 06:04:51.552092 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:51.551682    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s" event={"ID":"6a5b6319-91ea-449a-8196-0738360b4ec6","Type":"ContainerDied","Data":"58cf9949daa82df6c6d3f1cd8ce48d3c8779a5ff6e1936bd515cb5eee600669c"}
Nov 01 06:04:51.569160 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:51.569140    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s"
Nov 01 06:04:51.663182 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:51.663152    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a5b6319-91ea-449a-8196-0738360b4ec6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5b6319-91ea-449a-8196-0738360b4ec6-plugins-config-volume\") pod \"6a5b6319-91ea-449a-8196-0738360b4ec6\" (UID: \"6a5b6319-91ea-449a-8196-0738360b4ec6\") "
Nov 01 06:04:51.663372 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:51.663245    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5b6319-91ea-449a-8196-0738360b4ec6-kube-api-access-hz82d\" (UniqueName: \"kubernetes.io/projected/6a5b6319-91ea-449a-8196-0738360b4ec6-kube-api-access-hz82d\") pod \"6a5b6319-91ea-449a-8196-0738360b4ec6\" (UID: \"6a5b6319-91ea-449a-8196-0738360b4ec6\") "
Nov 01 06:04:51.663606 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:51.663581    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5b6319-91ea-449a-8196-0738360b4ec6-plugins-config-volume" pod "6a5b6319-91ea-449a-8196-0738360b4ec6" (UID: "6a5b6319-91ea-449a-8196-0738360b4ec6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:04:51.665333 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:51.665305    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5b6319-91ea-449a-8196-0738360b4ec6-kube-api-access-hz82d" pod "6a5b6319-91ea-449a-8196-0738360b4ec6" (UID: "6a5b6319-91ea-449a-8196-0738360b4ec6"). InnerVolumeSpecName "kube-api-access-hz82d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:04:51.764238 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:51.764208    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5b6319-91ea-449a-8196-0738360b4ec6-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:51.764238 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:51.764241    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz82d\" (UniqueName: \"kubernetes.io/projected/6a5b6319-91ea-449a-8196-0738360b4ec6-kube-api-access-hz82d\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:04:51.815766 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:51.815691    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:52.558455 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:52.558420    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s"
Nov 01 06:04:52.558455 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:52.558437    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s" event={"ID":"6a5b6319-91ea-449a-8196-0738360b4ec6","Type":"ContainerDied","Data":"224c8f8b824c149c733a57f832baf39338d5cb6177204c43ba0bf41110387e9d"}
Nov 01 06:04:52.558975 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:52.558502    5619 scope.go:122] "RemoveContainer" containerID="58cf9949daa82df6c6d3f1cd8ce48d3c8779a5ff6e1936bd515cb5eee600669c"
Nov 01 06:04:52.725578 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:52.725544    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s"]
Nov 01 06:04:52.776364 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:52.776333    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-58d523-7c54c9c87-tpc5s"]
Nov 01 06:04:52.964654 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:52.964621    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a5b6319-91ea-449a-8196-0738360b4ec6" path="/var/lib/kubelet/pods/6a5b6319-91ea-449a-8196-0738360b4ec6/volumes"
Nov 01 06:04:53.614335 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:53.614292    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:53.785537 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:53.785503    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:54.023054 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.022972    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg"]
Nov 01 06:04:54.023507 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.023493    5619 cpu_manager.go:414] "containerMap: removing container" podUID="6a5b6319-91ea-449a-8196-0738360b4ec6" containerName="payload-pre-processing"
Nov 01 06:04:54.023554 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.023508    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a5b6319-91ea-449a-8196-0738360b4ec6" containerName="payload-pre-processing"
Nov 01 06:04:54.023554 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.023528    5619 cpu_manager.go:414] "containerMap: removing container" podUID="2670ba1b-cdaa-48f4-9808-b34fa5632cbd" containerName="istio-proxy"
Nov 01 06:04:54.023554 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.023534    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2670ba1b-cdaa-48f4-9808-b34fa5632cbd" containerName="istio-proxy"
Nov 01 06:04:54.023649 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.023636    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="2670ba1b-cdaa-48f4-9808-b34fa5632cbd" containerName="istio-proxy"
Nov 01 06:04:54.023681 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.023650    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a5b6319-91ea-449a-8196-0738360b4ec6" containerName="payload-pre-processing"
Nov 01 06:04:54.028205 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.028188    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg"
Nov 01 06:04:54.074460 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.074432    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"]
Nov 01 06:04:54.078147 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.078125    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.088926 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.088904    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab0d2dd6-34d8-45fa-953b-b70f15835c55-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg\" (UID: \"ab0d2dd6-34d8-45fa-953b-b70f15835c55\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg"
Nov 01 06:04:54.089049 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.088975    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mlds\" (UniqueName: \"kubernetes.io/projected/ab0d2dd6-34d8-45fa-953b-b70f15835c55-kube-api-access-5mlds\") pod \"payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg\" (UID: \"ab0d2dd6-34d8-45fa-953b-b70f15835c55\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg"
Nov 01 06:04:54.089049 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.089004    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.089049 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.089024    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.089213 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.089096    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.089213 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.089144    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.089213 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.089192    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.089392 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.089224    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.089392 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.089288    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.089507 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.089388    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.089507 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.089473    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqbxp\" (UniqueName: \"kubernetes.io/projected/dfbc6deb-ab23-4706-b60e-2eb924fbb008-kube-api-access-bqbxp\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.100498 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.100468    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2d3f8d-dockercfg-z42wj\""
Nov 01 06:04:54.100498 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.100481    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2d3f8d\""
Nov 01 06:04:54.102794 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.102772    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg"]
Nov 01 06:04:54.190758 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.189237    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr"]
Nov 01 06:04:54.194273 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.194239    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqbxp\" (UniqueName: \"kubernetes.io/projected/dfbc6deb-ab23-4706-b60e-2eb924fbb008-kube-api-access-bqbxp\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.194885 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.194860    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab0d2dd6-34d8-45fa-953b-b70f15835c55-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg\" (UID: \"ab0d2dd6-34d8-45fa-953b-b70f15835c55\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg"
Nov 01 06:04:54.195059 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.195041    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mlds\" (UniqueName: \"kubernetes.io/projected/ab0d2dd6-34d8-45fa-953b-b70f15835c55-kube-api-access-5mlds\") pod \"payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg\" (UID: \"ab0d2dd6-34d8-45fa-953b-b70f15835c55\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg"
Nov 01 06:04:54.195191 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.195170    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.195309 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.195293    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.195463 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.195445    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.198281 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.198258    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.198472 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.198455    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.198592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.198578    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.198793 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.198777    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.198959 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.198945    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.199916 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.199477    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-data\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.201250 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.195739    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr"
Nov 01 06:04:54.201614 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.198173    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istiod-ca-cert\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.201614 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.197519    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-credential-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.201614 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.197167    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-workload-socket\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.201614 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.200816    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-workload-certs\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.201947 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.196871    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab0d2dd6-34d8-45fa-953b-b70f15835c55-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg\" (UID: \"ab0d2dd6-34d8-45fa-953b-b70f15835c55\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg"
Nov 01 06:04:54.223658 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.223627    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-envoy\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.231616 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.229627    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"]
Nov 01 06:04:54.231957 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.231921    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-podinfo\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.256796 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.256767    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqbxp\" (UniqueName: \"kubernetes.io/projected/dfbc6deb-ab23-4706-b60e-2eb924fbb008-kube-api-access-bqbxp\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.289408 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.287621    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr"]
Nov 01 06:04:54.289547 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.289517    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mlds\" (UniqueName: \"kubernetes.io/projected/ab0d2dd6-34d8-45fa-953b-b70f15835c55-kube-api-access-5mlds\") pod \"payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg\" (UID: \"ab0d2dd6-34d8-45fa-953b-b70f15835c55\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg"
Nov 01 06:04:54.291886 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.291822    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-token\") pod \"e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") " pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.300671 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.300631    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bad5155-b5e5-4352-93be-5a3230c62cf1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr\" (UID: \"8bad5155-b5e5-4352-93be-5a3230c62cf1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr"
Nov 01 06:04:54.300884 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.300861    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42pqr\" (UniqueName: \"kubernetes.io/projected/8bad5155-b5e5-4352-93be-5a3230c62cf1-kube-api-access-42pqr\") pod \"payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr\" (UID: \"8bad5155-b5e5-4352-93be-5a3230c62cf1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr"
Nov 01 06:04:54.341123 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.341094    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg"
Nov 01 06:04:54.395341 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.394490    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:54.403636 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.403602    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42pqr\" (UniqueName: \"kubernetes.io/projected/8bad5155-b5e5-4352-93be-5a3230c62cf1-kube-api-access-42pqr\") pod \"payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr\" (UID: \"8bad5155-b5e5-4352-93be-5a3230c62cf1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr"
Nov 01 06:04:54.403778 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.403759    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bad5155-b5e5-4352-93be-5a3230c62cf1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr\" (UID: \"8bad5155-b5e5-4352-93be-5a3230c62cf1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr"
Nov 01 06:04:54.404595 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.404556    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bad5155-b5e5-4352-93be-5a3230c62cf1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr\" (UID: \"8bad5155-b5e5-4352-93be-5a3230c62cf1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr"
Nov 01 06:04:54.469367 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.469190    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42pqr\" (UniqueName: \"kubernetes.io/projected/8bad5155-b5e5-4352-93be-5a3230c62cf1-kube-api-access-42pqr\") pod \"payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr\" (UID: \"8bad5155-b5e5-4352-93be-5a3230c62cf1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr"
Nov 01 06:04:54.557746 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.557667    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr"
Nov 01 06:04:54.724197 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.724173    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg"]
Nov 01 06:04:54.726186 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:04:54.726158    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0d2dd6_34d8_45fa_953b_b70f15835c55.slice/crio-1066abc94c9d3aaab0091757488061c14a328344bbaec9c41339fa7bc664aea7 WatchSource:0}: Error finding container 1066abc94c9d3aaab0091757488061c14a328344bbaec9c41339fa7bc664aea7: Status 404 returned error can't find the container with id 1066abc94c9d3aaab0091757488061c14a328344bbaec9c41339fa7bc664aea7
Nov 01 06:04:54.809040 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.808986    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"]
Nov 01 06:04:54.809736 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:04:54.809698    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfbc6deb_ab23_4706_b60e_2eb924fbb008.slice/crio-93e31a67c5e6fc2c3175cfb22d451209fc7a44329fedf1568cbcbc44cf928de9 WatchSource:0}: Error finding container 93e31a67c5e6fc2c3175cfb22d451209fc7a44329fedf1568cbcbc44cf928de9: Status 404 returned error can't find the container with id 93e31a67c5e6fc2c3175cfb22d451209fc7a44329fedf1568cbcbc44cf928de9
Nov 01 06:04:54.812067 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.812036    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:04:54.812163 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.812112    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:04:54.812163 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.812155    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:04:54.939430 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:54.939403    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr"]
Nov 01 06:04:54.941113 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:04:54.941081    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bad5155_b5e5_4352_93be_5a3230c62cf1.slice/crio-78f1d036f8195eb8733d79c4406a2da166570ff90fdb3108879e6b472703e618 WatchSource:0}: Error finding container 78f1d036f8195eb8733d79c4406a2da166570ff90fdb3108879e6b472703e618: Status 404 returned error can't find the container with id 78f1d036f8195eb8733d79c4406a2da166570ff90fdb3108879e6b472703e618
Nov 01 06:04:55.595096 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:55.595036    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr" event={"ID":"8bad5155-b5e5-4352-93be-5a3230c62cf1","Type":"ContainerStarted","Data":"795290df61a6f139e95deddbd81912aeae36af30fb15226fb5326ebe8198e86c"}
Nov 01 06:04:55.595295 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:55.595148    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr" event={"ID":"8bad5155-b5e5-4352-93be-5a3230c62cf1","Type":"ContainerStarted","Data":"78f1d036f8195eb8733d79c4406a2da166570ff90fdb3108879e6b472703e618"}
Nov 01 06:04:55.599401 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:55.597587    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2" event={"ID":"dfbc6deb-ab23-4706-b60e-2eb924fbb008","Type":"ContainerStarted","Data":"c7f052316e9f6ca44265b3eb5ff0153e88ac7f51b5264fea791c0c13c6835ad1"}
Nov 01 06:04:55.599401 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:55.597650    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2" event={"ID":"dfbc6deb-ab23-4706-b60e-2eb924fbb008","Type":"ContainerStarted","Data":"93e31a67c5e6fc2c3175cfb22d451209fc7a44329fedf1568cbcbc44cf928de9"}
Nov 01 06:04:55.601690 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:55.599879    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg" event={"ID":"ab0d2dd6-34d8-45fa-953b-b70f15835c55","Type":"ContainerStarted","Data":"2ac52a79b6d01dd1df7b6d157bc6a8561ecdca7793d50f3a7ffdaf02f550f297"}
Nov 01 06:04:55.601690 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:55.599956    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg" event={"ID":"ab0d2dd6-34d8-45fa-953b-b70f15835c55","Type":"ContainerStarted","Data":"1066abc94c9d3aaab0091757488061c14a328344bbaec9c41339fa7bc664aea7"}
Nov 01 06:04:55.601690 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:55.600189    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg"
Nov 01 06:04:55.714403 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:55.714284    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr" podStartSLOduration=1.714270465 podStartE2EDuration="1.714270465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:04:55.712491924 +0000 UTC m=+1132.923315432" watchObservedRunningTime="2026-11-01 06:04:55.714270465 +0000 UTC m=+1132.925093974"
Nov 01 06:04:55.840647 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:55.840574    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2" podStartSLOduration=1.8405470309999998 podStartE2EDuration="1.840547031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:04:55.836178058 +0000 UTC m=+1133.047001566" watchObservedRunningTime="2026-11-01 06:04:55.840547031 +0000 UTC m=+1133.051370560"
Nov 01 06:04:55.910009 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:04:55.909971    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:04:56.416887 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:56.416432    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:04:56.417295 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:56.417271    5619 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 01 06:04:56.417517 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:56.417489    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2" podUID="dfbc6deb-ab23-4706-b60e-2eb924fbb008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 01 06:04:56.606179 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:56.606143    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr"
Nov 01 06:04:56.906082 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:56.906043    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:04:57.395600 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:57.395560    5619 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 01 06:04:57.395785 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:57.395624    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2" podUID="dfbc6deb-ab23-4706-b60e-2eb924fbb008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 01 06:04:58.395173 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:58.395139    5619 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 01 06:04:58.395678 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:58.395206    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2" podUID="dfbc6deb-ab23-4706-b60e-2eb924fbb008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 01 06:04:59.395539 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:59.395503    5619 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 01 06:04:59.395952 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:04:59.395558    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2" podUID="dfbc6deb-ab23-4706-b60e-2eb924fbb008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 01 06:05:00.395271 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:00.395241    5619 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 01 06:05:00.395482 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:00.395373    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2" podUID="dfbc6deb-ab23-4706-b60e-2eb924fbb008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 01 06:05:01.394970 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:01.394930    5619 patch_prober.go:29] interesting pod/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 01 06:05:01.395412 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:01.395009    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2" podUID="dfbc6deb-ab23-4706-b60e-2eb924fbb008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 01 06:05:01.848401 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:01.848362    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:02.400175 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:02.400141    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:05:02.400651 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:02.400457    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:05:02.401237 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:02.401218    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:05:02.573844 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:02.573794    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg" podStartSLOduration=9.57377962 podStartE2EDuration="9.57377962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:04:55.971661633 +0000 UTC m=+1133.182485146" watchObservedRunningTime="2026-11-01 06:05:02.57377962 +0000 UTC m=+1139.784603129"
Nov 01 06:05:04.193911 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.193877    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"]
Nov 01 06:05:04.199162 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.199143    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.350948 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.350903    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.350948 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.350943    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.351176 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.351041    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.351176 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.351155    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.351289 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.351203    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.351289 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.351237    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.351422 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.351402    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.351545 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.351518    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.351622 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.351610    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg78x\" (UniqueName: \"kubernetes.io/projected/929d43c9-ec49-4988-a7af-ad7f49fea0ff-kube-api-access-sg78x\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.406046 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.406000    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"]
Nov 01 06:05:04.453480 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.453375    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg78x\" (UniqueName: \"kubernetes.io/projected/929d43c9-ec49-4988-a7af-ad7f49fea0ff-kube-api-access-sg78x\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.453480 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.453422    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.453480 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.453448    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.453812 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.453483    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.453812 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.453532    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.453812 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.453562    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.453812 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.453593    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.453812 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.453666    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.453812 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.453790    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.454111 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.453820    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.454413 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.454233    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.454559 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.454494    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.454807 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.454787    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.454922 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.454819    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.456523 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.456474    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.456845 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.456829    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.515866 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.515834    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.516013 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.515971    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg78x\" (UniqueName: \"kubernetes.io/projected/929d43c9-ec49-4988-a7af-ad7f49fea0ff-kube-api-access-sg78x\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:04.812557 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:04.812458    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:05.250728 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:05.249634    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"]
Nov 01 06:05:05.254133 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:05:05.254082    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929d43c9_ec49_4988_a7af_ad7f49fea0ff.slice/crio-8d72ce0aeef59fe3b501fb259c0d57ef77b55d85e19bf982c5834fcfcd8f155d WatchSource:0}: Error finding container 8d72ce0aeef59fe3b501fb259c0d57ef77b55d85e19bf982c5834fcfcd8f155d: Status 404 returned error can't find the container with id 8d72ce0aeef59fe3b501fb259c0d57ef77b55d85e19bf982c5834fcfcd8f155d
Nov 01 06:05:05.258378 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:05.258342    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:05:05.258497 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:05.258428    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:05:05.258497 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:05.258476    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:05:05.667572 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:05.667535    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg" event={"ID":"929d43c9-ec49-4988-a7af-ad7f49fea0ff","Type":"ContainerStarted","Data":"c0bb602ee87978d53259695eb806cbffd2697611b51cdf51d48f5a252e2ad679"}
Nov 01 06:05:05.667748 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:05.667598    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg" event={"ID":"929d43c9-ec49-4988-a7af-ad7f49fea0ff","Type":"ContainerStarted","Data":"8d72ce0aeef59fe3b501fb259c0d57ef77b55d85e19bf982c5834fcfcd8f155d"}
Nov 01 06:05:05.814395 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:05.813822    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:05.844041 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:05.843980    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg" podStartSLOduration=1.843968028 podStartE2EDuration="1.843968028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:05:05.839873835 +0000 UTC m=+1143.050697356" watchObservedRunningTime="2026-11-01 06:05:05.843968028 +0000 UTC m=+1143.054791537"
Nov 01 06:05:06.586937 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:06.586898    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:06.609603 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:06.609573    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg"
Nov 01 06:05:06.814921 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:06.814888    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 01 06:05:06.815096 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:06.814976    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg" podUID="929d43c9-ec49-4988-a7af-ad7f49fea0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 01 06:05:07.615594 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:07.615565    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr"
Nov 01 06:05:07.813539 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:07.813499    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 01 06:05:07.813697 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:07.813559    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg" podUID="929d43c9-ec49-4988-a7af-ad7f49fea0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 01 06:05:08.813677 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:08.813643    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 01 06:05:08.814059 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:08.813694    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg" podUID="929d43c9-ec49-4988-a7af-ad7f49fea0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 01 06:05:09.813410 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:09.813378    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 01 06:05:09.813601 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:09.813454    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg" podUID="929d43c9-ec49-4988-a7af-ad7f49fea0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 01 06:05:10.813049 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:10.813009    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 01 06:05:10.813484 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:10.813066    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg" podUID="929d43c9-ec49-4988-a7af-ad7f49fea0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 01 06:05:11.502345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:11.501898    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:11.813671 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:11.813588    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 01 06:05:11.814145 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:11.813650    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg" podUID="929d43c9-ec49-4988-a7af-ad7f49fea0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 01 06:05:12.829347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:12.828663    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:12.829856 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:12.829607    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:12.832372 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:12.831312    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:05:16.379699 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:16.379663    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:17.781887 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:17.781850    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:20.786420 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:20.786306    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:21.381198 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:21.381161    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:24.881973 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:24.881937    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:25.361279 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:25.361249    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:25.881111 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:05:25.881080    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:05:27.986440 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:27.986317    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:33.375870 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:33.375829    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:34.317761 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:34.317724    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:38.041264 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:38.041230    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:38.200471 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:38.200439    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:38.355503 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:38.355460    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:42.964391 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:42.964272    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:47.789730 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:47.789692    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:52.575591 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:52.575542    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:54.465599 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:54.465562    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:54.965473 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:54.965436    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:55.881305 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:05:55.881265    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:05:55.960722 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:55.960680    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:56.153702 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:56.153613    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:05:57.399022 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:05:57.398990    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:00.251482 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.251444    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"]
Nov 01 06:06:00.258468 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.258439    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"
Nov 01 06:06:00.298226 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.298197    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4\" (UID: \"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"
Nov 01 06:06:00.298378 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.298255    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6g46\" (UniqueName: \"kubernetes.io/projected/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-kube-api-access-q6g46\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4\" (UID: \"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"
Nov 01 06:06:00.298433 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.298387    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4\" (UID: \"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"
Nov 01 06:06:00.298433 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.298428    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4\" (UID: \"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"
Nov 01 06:06:00.312925 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.312900    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"]
Nov 01 06:06:00.360528 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.360499    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"openshift-service-ca.crt\""
Nov 01 06:06:00.360665 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.360488    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"routing-overlay\""
Nov 01 06:06:00.360665 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.360516    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"agc-cred-28460a5a0934\""
Nov 01 06:06:00.360665 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.360556    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"payload-processing-external-model-plugins-e2e-extmod-b-3986bf\""
Nov 01 06:06:00.362622 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.362597    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"payload-processing-external-model-e2e--cf3b8510-dockercfg-ftjt9\""
Nov 01 06:06:00.362622 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.362608    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"kube-root-ca.crt\""
Nov 01 06:06:00.362745 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.362609    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"agc-cred-3feee30ca48b\""
Nov 01 06:06:00.399950 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.399921    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4\" (UID: \"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"
Nov 01 06:06:00.400064 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.399960    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4\" (UID: \"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"
Nov 01 06:06:00.400064 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.400061    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4\" (UID: \"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"
Nov 01 06:06:00.400160 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.400109    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6g46\" (UniqueName: \"kubernetes.io/projected/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-kube-api-access-q6g46\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4\" (UID: \"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"
Nov 01 06:06:00.400770 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.400744    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4\" (UID: \"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"
Nov 01 06:06:00.400770 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.400767    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4\" (UID: \"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"
Nov 01 06:06:00.402438 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.402416    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4\" (UID: \"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"
Nov 01 06:06:00.462225 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.462195    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6g46\" (UniqueName: \"kubernetes.io/projected/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-kube-api-access-q6g46\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4\" (UID: \"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"
Nov 01 06:06:00.552392 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.552306    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"]
Nov 01 06:06:00.556527 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.556508    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.573297 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.573269    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"
Nov 01 06:06:00.602718 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.602674    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-envoy\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.602874 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.602803    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-workload-certs\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.602938 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.602892    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-podinfo\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.602938 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.602928    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-workload-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.603040 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.602969    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-token\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.603040 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.603030    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5087163-8f0a-407b-b2be-cb9f9c888706-istiod-ca-cert\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.603149 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.603104    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgnld\" (UniqueName: \"kubernetes.io/projected/e5087163-8f0a-407b-b2be-cb9f9c888706-kube-api-access-rgnld\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.603216 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.603155    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-data\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.603216 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.603199    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-credential-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.618731 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.618698    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"]
Nov 01 06:06:00.695893 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.695859    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-2d3f8d-openshift-default-dockercfg-zw2df\""
Nov 01 06:06:00.705189 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.705150    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-envoy\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.705350 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.705257    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-workload-certs\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.705419 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.705366    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-podinfo\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.705478 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.705422    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-workload-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.705537 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.705486    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-token\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.705589 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.705536    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5087163-8f0a-407b-b2be-cb9f9c888706-istiod-ca-cert\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.705589 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.705570    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgnld\" (UniqueName: \"kubernetes.io/projected/e5087163-8f0a-407b-b2be-cb9f9c888706-kube-api-access-rgnld\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.705693 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.705605    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-data\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.705693 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.705665    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-credential-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.705903 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.705877    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-workload-certs\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.706075 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.706050    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-credential-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.706483 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.706456    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-workload-socket\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.707202 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.707177    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5087163-8f0a-407b-b2be-cb9f9c888706-istiod-ca-cert\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.707311 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.707226    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-data\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.708676 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.708648    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-podinfo\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.710902 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.710884    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-envoy\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.768964 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.768934    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-token\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.769119 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.769016    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgnld\" (UniqueName: \"kubernetes.io/projected/e5087163-8f0a-407b-b2be-cb9f9c888706-kube-api-access-rgnld\") pod \"e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") " pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.870580 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.870545    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:00.967220 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:00.967191    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"]
Nov 01 06:06:00.971540 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:06:00.971507    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9e6d41_5c7c_4523_94fb_16d3b0bd263a.slice/crio-5ec99da135b3aefd3f6c77c9ce6ed73f18756338f67eb5bcada0956dcffa578b WatchSource:0}: Error finding container 5ec99da135b3aefd3f6c77c9ce6ed73f18756338f67eb5bcada0956dcffa578b: Status 404 returned error can't find the container with id 5ec99da135b3aefd3f6c77c9ce6ed73f18756338f67eb5bcada0956dcffa578b
Nov 01 06:06:01.156243 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:01.156159    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4" event={"ID":"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a","Type":"ContainerStarted","Data":"c8aceb7d7cb5478e055aed6659dad3dab6a9c2223d8cff84aea778d629e36208"}
Nov 01 06:06:01.156243 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:01.156208    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4" event={"ID":"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a","Type":"ContainerStarted","Data":"5ec99da135b3aefd3f6c77c9ce6ed73f18756338f67eb5bcada0956dcffa578b"}
Nov 01 06:06:01.156455 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:01.156283    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"
Nov 01 06:06:01.290094 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:01.290067    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"]
Nov 01 06:06:01.290892 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:06:01.290865    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5087163_8f0a_407b_b2be_cb9f9c888706.slice/crio-0ad740fe6381ee32370c62b9de6208111d558690631d614ddaed3c7dd19f606a WatchSource:0}: Error finding container 0ad740fe6381ee32370c62b9de6208111d558690631d614ddaed3c7dd19f606a: Status 404 returned error can't find the container with id 0ad740fe6381ee32370c62b9de6208111d558690631d614ddaed3c7dd19f606a
Nov 01 06:06:01.292281 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:01.292228    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4" podStartSLOduration=1.292210174 podStartE2EDuration="1.292210174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:06:01.287930929 +0000 UTC m=+1198.498754440" watchObservedRunningTime="2026-11-01 06:06:01.292210174 +0000 UTC m=+1198.503033685"
Nov 01 06:06:01.293761 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:01.293730    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:06:01.293842 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:01.293791    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:06:01.293842 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:01.293822    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:06:02.179703 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:02.179664    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn" event={"ID":"e5087163-8f0a-407b-b2be-cb9f9c888706","Type":"ContainerStarted","Data":"791ac794f7c6cd9238eb82e307ef868cf6e5bd53c285ad4a6ce170229b4459d0"}
Nov 01 06:06:02.179907 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:02.179719    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn" event={"ID":"e5087163-8f0a-407b-b2be-cb9f9c888706","Type":"ContainerStarted","Data":"0ad740fe6381ee32370c62b9de6208111d558690631d614ddaed3c7dd19f606a"}
Nov 01 06:06:02.276098 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:02.276059    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:02.304197 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:02.304145    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn" podStartSLOduration=2.304131196 podStartE2EDuration="2.304131196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:06:02.302023278 +0000 UTC m=+1199.512846791" watchObservedRunningTime="2026-11-01 06:06:02.304131196 +0000 UTC m=+1199.514954704"
Nov 01 06:06:02.874448 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:02.874413    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:02.874963 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:02.874933    5619 patch_prober.go:29] interesting pod/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 01 06:06:02.875094 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:02.874987    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn" podUID="e5087163-8f0a-407b-b2be-cb9f9c888706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 01 06:06:03.873346 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:03.872703    5619 patch_prober.go:29] interesting pod/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 01 06:06:03.873346 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:03.872774    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn" podUID="e5087163-8f0a-407b-b2be-cb9f9c888706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 01 06:06:04.874347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:04.872714    5619 patch_prober.go:29] interesting pod/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 01 06:06:04.874347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:04.872777    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn" podUID="e5087163-8f0a-407b-b2be-cb9f9c888706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 01 06:06:05.873346 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:05.872658    5619 patch_prober.go:29] interesting pod/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 01 06:06:05.873346 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:05.872731    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn" podUID="e5087163-8f0a-407b-b2be-cb9f9c888706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 01 06:06:06.872345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:06.871664    5619 patch_prober.go:29] interesting pod/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 01 06:06:06.872345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:06.871724    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn" podUID="e5087163-8f0a-407b-b2be-cb9f9c888706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 01 06:06:07.136894 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:07.136811    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:07.875373 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:07.875311    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:07.875851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:07.875631    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:07.876292 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:07.876273    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:06:11.050132 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:11.050086    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"]
Nov 01 06:06:11.153350 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:11.147952    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:12.182631 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:12.182605    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"
Nov 01 06:06:12.494779 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:12.494697    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:18.135372 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:18.135305    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:18.330361 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:18.330303    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:18.969623 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:18.969590    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:19.751371 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:19.751298    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:19.943723 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:19.943676    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:22.467563 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:22.467526    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:23.657850 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:23.657811    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"]
Nov 01 06:06:23.662541 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:23.662510    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"
Nov 01 06:06:23.728373 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:23.717782    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"agc-cred-d344d6f1c604\""
Nov 01 06:06:23.746387 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:23.746348    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"]
Nov 01 06:06:23.756974 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:23.756941    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:23.802062 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:23.802028    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59595a9-a730-43f0-a4cd-c667b99f57d6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7\" (UID: \"e59595a9-a730-43f0-a4cd-c667b99f57d6\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"
Nov 01 06:06:23.802353 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:23.802315    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e59595a9-a730-43f0-a4cd-c667b99f57d6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7\" (UID: \"e59595a9-a730-43f0-a4cd-c667b99f57d6\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"
Nov 01 06:06:23.802591 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:23.802571    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x46bv\" (UniqueName: \"kubernetes.io/projected/e59595a9-a730-43f0-a4cd-c667b99f57d6-kube-api-access-x46bv\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7\" (UID: \"e59595a9-a730-43f0-a4cd-c667b99f57d6\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"
Nov 01 06:06:23.802912 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:23.802890    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e59595a9-a730-43f0-a4cd-c667b99f57d6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7\" (UID: \"e59595a9-a730-43f0-a4cd-c667b99f57d6\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"
Nov 01 06:06:23.903991 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:23.903954    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59595a9-a730-43f0-a4cd-c667b99f57d6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7\" (UID: \"e59595a9-a730-43f0-a4cd-c667b99f57d6\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"
Nov 01 06:06:23.904197 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:23.904015    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e59595a9-a730-43f0-a4cd-c667b99f57d6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7\" (UID: \"e59595a9-a730-43f0-a4cd-c667b99f57d6\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"
Nov 01 06:06:23.904197 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:23.904045    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x46bv\" (UniqueName: \"kubernetes.io/projected/e59595a9-a730-43f0-a4cd-c667b99f57d6-kube-api-access-x46bv\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7\" (UID: \"e59595a9-a730-43f0-a4cd-c667b99f57d6\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"
Nov 01 06:06:23.904197 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:23.904138    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e59595a9-a730-43f0-a4cd-c667b99f57d6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7\" (UID: \"e59595a9-a730-43f0-a4cd-c667b99f57d6\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"
Nov 01 06:06:23.904822 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:23.904783    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59595a9-a730-43f0-a4cd-c667b99f57d6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7\" (UID: \"e59595a9-a730-43f0-a4cd-c667b99f57d6\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"
Nov 01 06:06:23.904975 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:23.904856    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e59595a9-a730-43f0-a4cd-c667b99f57d6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7\" (UID: \"e59595a9-a730-43f0-a4cd-c667b99f57d6\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"
Nov 01 06:06:23.906915 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:23.906891    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e59595a9-a730-43f0-a4cd-c667b99f57d6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7\" (UID: \"e59595a9-a730-43f0-a4cd-c667b99f57d6\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"
Nov 01 06:06:23.966361 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:23.966262    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x46bv\" (UniqueName: \"kubernetes.io/projected/e59595a9-a730-43f0-a4cd-c667b99f57d6-kube-api-access-x46bv\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7\" (UID: \"e59595a9-a730-43f0-a4cd-c667b99f57d6\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"
Nov 01 06:06:23.979946 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:23.979924    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"
Nov 01 06:06:24.358523 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:24.358499    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"]
Nov 01 06:06:24.359912 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:06:24.359885    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59595a9_a730_43f0_a4cd_c667b99f57d6.slice/crio-c14f7598248d7139208a8c6b5d4ff2917a13f096ae81ed467750a9a1a364d807 WatchSource:0}: Error finding container c14f7598248d7139208a8c6b5d4ff2917a13f096ae81ed467750a9a1a364d807: Status 404 returned error can't find the container with id c14f7598248d7139208a8c6b5d4ff2917a13f096ae81ed467750a9a1a364d807
Nov 01 06:06:25.363463 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:25.363424    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7" event={"ID":"e59595a9-a730-43f0-a4cd-c667b99f57d6","Type":"ContainerStarted","Data":"93c2f3fe9d5c55bb8793e0622633a7d82aa5e47e03aca034c43b28bed7287eef"}
Nov 01 06:06:25.363851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:25.363489    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7" event={"ID":"e59595a9-a730-43f0-a4cd-c667b99f57d6","Type":"ContainerStarted","Data":"c14f7598248d7139208a8c6b5d4ff2917a13f096ae81ed467750a9a1a364d807"}
Nov 01 06:06:25.363851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:25.363519    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"
Nov 01 06:06:25.481977 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:25.481918    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7" podStartSLOduration=2.481900715 podStartE2EDuration="2.481900715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:06:25.481148408 +0000 UTC m=+1222.691971928" watchObservedRunningTime="2026-11-01 06:06:25.481900715 +0000 UTC m=+1222.692724225"
Nov 01 06:06:25.884823 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:06:25.884783    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:06:26.274840 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:26.274743    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6" podUID="0bc76ed5-591e-4230-8c24-8c199d46d87e" containerName="main" containerID="cri-o://b56e63713bbc2c7881fb7a6fb60e85eb83a3de5ac584398b65f4eafb635ac030" gracePeriod=45
Nov 01 06:06:26.325740 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:26.325712    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:26.657982 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:26.657953    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:06:26.741083 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:26.741052    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bc76ed5-591e-4230-8c24-8c199d46d87e-kube-api-access-hrvmn\" (UniqueName: \"kubernetes.io/projected/0bc76ed5-591e-4230-8c24-8c199d46d87e-kube-api-access-hrvmn\") pod \"0bc76ed5-591e-4230-8c24-8c199d46d87e\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") "
Nov 01 06:06:26.741247 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:26.741130    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-home\" (UniqueName: \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-home\") pod \"0bc76ed5-591e-4230-8c24-8c199d46d87e\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") "
Nov 01 06:06:26.741247 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:26.741210    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-dshm\") pod \"0bc76ed5-591e-4230-8c24-8c199d46d87e\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") "
Nov 01 06:06:26.741247 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:26.741239    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-tmp-dir\") pod \"0bc76ed5-591e-4230-8c24-8c199d46d87e\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") "
Nov 01 06:06:26.741464 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:26.741293    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-model-cache\") pod \"0bc76ed5-591e-4230-8c24-8c199d46d87e\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") "
Nov 01 06:06:26.741464 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:26.741387    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bc76ed5-591e-4230-8c24-8c199d46d87e-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc76ed5-591e-4230-8c24-8c199d46d87e-tls-certs\") pod \"0bc76ed5-591e-4230-8c24-8c199d46d87e\" (UID: \"0bc76ed5-591e-4230-8c24-8c199d46d87e\") "
Nov 01 06:06:26.741464 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:26.741408    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-home" pod "0bc76ed5-591e-4230-8c24-8c199d46d87e" (UID: "0bc76ed5-591e-4230-8c24-8c199d46d87e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:06:26.741619 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:26.741545    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-tmp-dir" pod "0bc76ed5-591e-4230-8c24-8c199d46d87e" (UID: "0bc76ed5-591e-4230-8c24-8c199d46d87e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:06:26.741619 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:26.741578    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-model-cache" pod "0bc76ed5-591e-4230-8c24-8c199d46d87e" (UID: "0bc76ed5-591e-4230-8c24-8c199d46d87e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:06:26.741936 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:26.741918    5619 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-home\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:06:26.741936 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:26.741936    5619 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-tmp-dir\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:06:26.742126 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:26.741945    5619 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-model-cache\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:06:26.743484 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:26.743456    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-dshm" pod "0bc76ed5-591e-4230-8c24-8c199d46d87e" (UID: "0bc76ed5-591e-4230-8c24-8c199d46d87e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:06:26.743583 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:26.743549    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bc76ed5-591e-4230-8c24-8c199d46d87e-tls-certs" pod "0bc76ed5-591e-4230-8c24-8c199d46d87e" (UID: "0bc76ed5-591e-4230-8c24-8c199d46d87e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:06:26.743583 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:26.743557    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc76ed5-591e-4230-8c24-8c199d46d87e-kube-api-access-hrvmn" pod "0bc76ed5-591e-4230-8c24-8c199d46d87e" (UID: "0bc76ed5-591e-4230-8c24-8c199d46d87e"). InnerVolumeSpecName "kube-api-access-hrvmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:06:26.842965 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:26.842926    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrvmn\" (UniqueName: \"kubernetes.io/projected/0bc76ed5-591e-4230-8c24-8c199d46d87e-kube-api-access-hrvmn\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:06:26.842965 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:26.842966    5619 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc76ed5-591e-4230-8c24-8c199d46d87e-dshm\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:06:26.843162 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:26.842978    5619 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc76ed5-591e-4230-8c24-8c199d46d87e-tls-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:06:27.375668 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.375633    5619 generic.go:356] "Generic (PLEG): container finished" podID="0bc76ed5-591e-4230-8c24-8c199d46d87e" containerID="b56e63713bbc2c7881fb7a6fb60e85eb83a3de5ac584398b65f4eafb635ac030" exitCode=0
Nov 01 06:06:27.375913 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.375674    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"
Nov 01 06:06:27.375913 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.375709    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6" event={"ID":"0bc76ed5-591e-4230-8c24-8c199d46d87e","Type":"ContainerDied","Data":"b56e63713bbc2c7881fb7a6fb60e85eb83a3de5ac584398b65f4eafb635ac030"}
Nov 01 06:06:27.375913 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.375750    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6" event={"ID":"0bc76ed5-591e-4230-8c24-8c199d46d87e","Type":"ContainerDied","Data":"890b97d12f5eae0de7246fe7e212a23f4e91001a96f14f571286f320b38cda59"}
Nov 01 06:06:27.375913 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.375775    5619 scope.go:122] "RemoveContainer" containerID="b56e63713bbc2c7881fb7a6fb60e85eb83a3de5ac584398b65f4eafb635ac030"
Nov 01 06:06:27.385981 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.385962    5619 scope.go:122] "RemoveContainer" containerID="b56e63713bbc2c7881fb7a6fb60e85eb83a3de5ac584398b65f4eafb635ac030"
Nov 01 06:06:27.386236 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:06:27.386213    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b56e63713bbc2c7881fb7a6fb60e85eb83a3de5ac584398b65f4eafb635ac030\": container with ID starting with b56e63713bbc2c7881fb7a6fb60e85eb83a3de5ac584398b65f4eafb635ac030 not found: ID does not exist" containerID="b56e63713bbc2c7881fb7a6fb60e85eb83a3de5ac584398b65f4eafb635ac030"
Nov 01 06:06:27.386312 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.386252    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56e63713bbc2c7881fb7a6fb60e85eb83a3de5ac584398b65f4eafb635ac030"} err="failed to get container status \"b56e63713bbc2c7881fb7a6fb60e85eb83a3de5ac584398b65f4eafb635ac030\": rpc error: code = NotFound desc = could not find container \"b56e63713bbc2c7881fb7a6fb60e85eb83a3de5ac584398b65f4eafb635ac030\": container with ID starting with b56e63713bbc2c7881fb7a6fb60e85eb83a3de5ac584398b65f4eafb635ac030 not found: ID does not exist"
Nov 01 06:06:27.483479 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.483450    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"]
Nov 01 06:06:27.484071 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.484052    5619 cpu_manager.go:414] "containerMap: removing container" podUID="0bc76ed5-591e-4230-8c24-8c199d46d87e" containerName="main"
Nov 01 06:06:27.484167 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.484073    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bc76ed5-591e-4230-8c24-8c199d46d87e" containerName="main"
Nov 01 06:06:27.484271 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.484256    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bc76ed5-591e-4230-8c24-8c199d46d87e" containerName="main"
Nov 01 06:06:27.489065 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.489044    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"
Nov 01 06:06:27.539394 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.539350    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"]
Nov 01 06:06:27.544206 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.544179    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"]
Nov 01 06:06:27.544527 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.544507    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cfe368\"/\"openshift-service-ca.crt\""
Nov 01 06:06:27.544640 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.544624    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cfe368\"/\"routing-overlay\""
Nov 01 06:06:27.544735 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.544699    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cfe368\"/\"payload-processing-external-model-plugins-e2e-extmod-a-cfe368\""
Nov 01 06:06:27.544798 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.544767    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cfe368\"/\"agc-cred-83f5507c1f28\""
Nov 01 06:06:27.544927 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.544893    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cfe368\"/\"payload-processing-external-model-e2e--6d943d76-dockercfg-gr87h\""
Nov 01 06:06:27.545043 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.544969    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cfe368\"/\"kube-root-ca.crt\""
Nov 01 06:06:27.588915 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.588890    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0d2807/test-model-0d2807-kserve-69b99c649-22st6"]
Nov 01 06:06:27.653658 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.653579    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23b253a4-3245-4e42-a164-626dd69baf75-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p\" (UID: \"23b253a4-3245-4e42-a164-626dd69baf75\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"
Nov 01 06:06:27.653806 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.653745    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b253a4-3245-4e42-a164-626dd69baf75-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p\" (UID: \"23b253a4-3245-4e42-a164-626dd69baf75\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"
Nov 01 06:06:27.653893 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.653865    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23b253a4-3245-4e42-a164-626dd69baf75-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p\" (UID: \"23b253a4-3245-4e42-a164-626dd69baf75\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"
Nov 01 06:06:27.654266 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.653904    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bhn9\" (UniqueName: \"kubernetes.io/projected/23b253a4-3245-4e42-a164-626dd69baf75-kube-api-access-2bhn9\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p\" (UID: \"23b253a4-3245-4e42-a164-626dd69baf75\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"
Nov 01 06:06:27.754983 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.754931    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23b253a4-3245-4e42-a164-626dd69baf75-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p\" (UID: \"23b253a4-3245-4e42-a164-626dd69baf75\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"
Nov 01 06:06:27.755398 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.755058    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b253a4-3245-4e42-a164-626dd69baf75-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p\" (UID: \"23b253a4-3245-4e42-a164-626dd69baf75\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"
Nov 01 06:06:27.755398 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.755103    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23b253a4-3245-4e42-a164-626dd69baf75-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p\" (UID: \"23b253a4-3245-4e42-a164-626dd69baf75\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"
Nov 01 06:06:27.755398 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.755130    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bhn9\" (UniqueName: \"kubernetes.io/projected/23b253a4-3245-4e42-a164-626dd69baf75-kube-api-access-2bhn9\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p\" (UID: \"23b253a4-3245-4e42-a164-626dd69baf75\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"
Nov 01 06:06:27.755769 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.755746    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23b253a4-3245-4e42-a164-626dd69baf75-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p\" (UID: \"23b253a4-3245-4e42-a164-626dd69baf75\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"
Nov 01 06:06:27.755893 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.755872    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b253a4-3245-4e42-a164-626dd69baf75-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p\" (UID: \"23b253a4-3245-4e42-a164-626dd69baf75\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"
Nov 01 06:06:27.757606 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.757583    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23b253a4-3245-4e42-a164-626dd69baf75-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p\" (UID: \"23b253a4-3245-4e42-a164-626dd69baf75\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"
Nov 01 06:06:27.816879 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:27.816853    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bhn9\" (UniqueName: \"kubernetes.io/projected/23b253a4-3245-4e42-a164-626dd69baf75-kube-api-access-2bhn9\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p\" (UID: \"23b253a4-3245-4e42-a164-626dd69baf75\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"
Nov 01 06:06:28.102059 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:28.102030    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"
Nov 01 06:06:28.482304 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:28.482203    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:28.484656 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:28.484628    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"]
Nov 01 06:06:28.485637 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:06:28.485612    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b253a4_3245_4e42_a164_626dd69baf75.slice/crio-9d8216ccf5cafbbec5a7c91a24768183f198dbf79dedcaf99979bd6db273412f WatchSource:0}: Error finding container 9d8216ccf5cafbbec5a7c91a24768183f198dbf79dedcaf99979bd6db273412f: Status 404 returned error can't find the container with id 9d8216ccf5cafbbec5a7c91a24768183f198dbf79dedcaf99979bd6db273412f
Nov 01 06:06:28.965504 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:28.965465    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bc76ed5-591e-4230-8c24-8c199d46d87e" path="/var/lib/kubelet/pods/0bc76ed5-591e-4230-8c24-8c199d46d87e/volumes"
Nov 01 06:06:29.392206 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:29.392169    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p" event={"ID":"23b253a4-3245-4e42-a164-626dd69baf75","Type":"ContainerStarted","Data":"5f262be1ef193970bc9e1d46540554fba2e48d9a5d64bc3f5fbfab52af02a873"}
Nov 01 06:06:29.392451 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:29.392236    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p" event={"ID":"23b253a4-3245-4e42-a164-626dd69baf75","Type":"ContainerStarted","Data":"9d8216ccf5cafbbec5a7c91a24768183f198dbf79dedcaf99979bd6db273412f"}
Nov 01 06:06:29.392451 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:29.392386    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"
Nov 01 06:06:29.511472 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:29.511420    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p" podStartSLOduration=2.511404772 podStartE2EDuration="2.511404772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:06:29.509792916 +0000 UTC m=+1226.720616424" watchObservedRunningTime="2026-11-01 06:06:29.511404772 +0000 UTC m=+1226.722228281"
Nov 01 06:06:34.028349 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:34.026522    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:36.372428 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:36.372397    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"
Nov 01 06:06:36.525115 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:36.525077    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"]
Nov 01 06:06:36.682966 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:36.682877    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:37.948525 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:37.948494    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr"]
Nov 01 06:06:38.062410 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:38.062375    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg"]
Nov 01 06:06:38.311510 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:38.311432    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:38.510373 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:38.510337    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:39.185585 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:39.185554    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:40.403177 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:40.403141    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"
Nov 01 06:06:40.561563 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:06:40.561531    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-cfe368/agc-cred-83f5507c1f28: secret "agc-cred-83f5507c1f28" not found
Nov 01 06:06:40.561783 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:06:40.561759    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p: secret "agc-cred-83f5507c1f28" not found
Nov 01 06:06:40.561946 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:06:40.561850    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23b253a4-3245-4e42-a164-626dd69baf75-provider-credentials podName:23b253a4-3245-4e42-a164-626dd69baf75 nodeName:}" failed. No retries permitted until 2026-11-01 06:06:41.061826882 +0000 UTC m=+1238.272650386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/23b253a4-3245-4e42-a164-626dd69baf75-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p" (UID: "23b253a4-3245-4e42-a164-626dd69baf75") : secret "agc-cred-83f5507c1f28" not found
Nov 01 06:06:40.608707 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:40.608679    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"]
Nov 01 06:06:40.762117 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:40.762025    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:41.068643 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:06:41.068560    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-cfe368/agc-cred-83f5507c1f28: secret "agc-cred-83f5507c1f28" not found
Nov 01 06:06:41.068643 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:06:41.068583    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p: secret "agc-cred-83f5507c1f28" not found
Nov 01 06:06:41.068643 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:06:41.068643    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23b253a4-3245-4e42-a164-626dd69baf75-provider-credentials podName:23b253a4-3245-4e42-a164-626dd69baf75 nodeName:}" failed. No retries permitted until 2026-11-01 06:06:42.068622084 +0000 UTC m=+1239.279445599 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/23b253a4-3245-4e42-a164-626dd69baf75-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p" (UID: "23b253a4-3245-4e42-a164-626dd69baf75") : secret "agc-cred-83f5507c1f28" not found
Nov 01 06:06:41.525431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:41.525382    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4" podUID="6c9e6d41-5c7c-4523-94fb-16d3b0bd263a" containerName="payload-processing" containerID="cri-o://c8aceb7d7cb5478e055aed6659dad3dab6a9c2223d8cff84aea778d629e36208" gracePeriod=25
Nov 01 06:06:41.799270 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:41.799240    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"
Nov 01 06:06:41.881191 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:41.881156    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-routing-overlay-volume\") pod \"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a\" (UID: \"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a\") "
Nov 01 06:06:41.881401 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:41.881211    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-plugins-config-volume\") pod \"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a\" (UID: \"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a\") "
Nov 01 06:06:41.881401 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:41.881302    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-kube-api-access-q6g46\" (UniqueName: \"kubernetes.io/projected/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-kube-api-access-q6g46\") pod \"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a\" (UID: \"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a\") "
Nov 01 06:06:41.881543 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:41.881491    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-provider-credentials\") pod \"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a\" (UID: \"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a\") "
Nov 01 06:06:41.881659 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:41.881634    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-routing-overlay-volume" pod "6c9e6d41-5c7c-4523-94fb-16d3b0bd263a" (UID: "6c9e6d41-5c7c-4523-94fb-16d3b0bd263a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:06:41.881732 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:41.881645    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-plugins-config-volume" pod "6c9e6d41-5c7c-4523-94fb-16d3b0bd263a" (UID: "6c9e6d41-5c7c-4523-94fb-16d3b0bd263a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:06:41.881999 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:41.881969    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:06:41.882062 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:41.882005    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:06:41.883587 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:41.883562    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-provider-credentials" pod "6c9e6d41-5c7c-4523-94fb-16d3b0bd263a" (UID: "6c9e6d41-5c7c-4523-94fb-16d3b0bd263a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:06:41.883660 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:41.883599    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-kube-api-access-q6g46" pod "6c9e6d41-5c7c-4523-94fb-16d3b0bd263a" (UID: "6c9e6d41-5c7c-4523-94fb-16d3b0bd263a"). InnerVolumeSpecName "kube-api-access-q6g46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:06:41.983406 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:41.983364    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6g46\" (UniqueName: \"kubernetes.io/projected/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-kube-api-access-q6g46\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:06:41.983406 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:41.983401    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:06:42.084539 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:06:42.084455    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-cfe368/agc-cred-83f5507c1f28: secret "agc-cred-83f5507c1f28" not found
Nov 01 06:06:42.084539 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:06:42.084482    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p: secret "agc-cred-83f5507c1f28" not found
Nov 01 06:06:42.084727 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:06:42.084550    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23b253a4-3245-4e42-a164-626dd69baf75-provider-credentials podName:23b253a4-3245-4e42-a164-626dd69baf75 nodeName:}" failed. No retries permitted until 2026-11-01 06:06:44.084533855 +0000 UTC m=+1241.295357343 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/23b253a4-3245-4e42-a164-626dd69baf75-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p" (UID: "23b253a4-3245-4e42-a164-626dd69baf75") : secret "agc-cred-83f5507c1f28" not found
Nov 01 06:06:42.631156 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:42.631124    5619 generic.go:356] "Generic (PLEG): container finished" podID="6c9e6d41-5c7c-4523-94fb-16d3b0bd263a" containerID="c8aceb7d7cb5478e055aed6659dad3dab6a9c2223d8cff84aea778d629e36208" exitCode=0
Nov 01 06:06:42.631644 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:42.631170    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"
Nov 01 06:06:42.631644 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:42.631193    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4" event={"ID":"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a","Type":"ContainerDied","Data":"c8aceb7d7cb5478e055aed6659dad3dab6a9c2223d8cff84aea778d629e36208"}
Nov 01 06:06:42.631644 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:42.631236    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4" event={"ID":"6c9e6d41-5c7c-4523-94fb-16d3b0bd263a","Type":"ContainerDied","Data":"5ec99da135b3aefd3f6c77c9ce6ed73f18756338f67eb5bcada0956dcffa578b"}
Nov 01 06:06:42.631644 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:42.631257    5619 scope.go:122] "RemoveContainer" containerID="c8aceb7d7cb5478e055aed6659dad3dab6a9c2223d8cff84aea778d629e36208"
Nov 01 06:06:42.641442 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:42.641423    5619 scope.go:122] "RemoveContainer" containerID="c8aceb7d7cb5478e055aed6659dad3dab6a9c2223d8cff84aea778d629e36208"
Nov 01 06:06:42.641717 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:06:42.641690    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8aceb7d7cb5478e055aed6659dad3dab6a9c2223d8cff84aea778d629e36208\": container with ID starting with c8aceb7d7cb5478e055aed6659dad3dab6a9c2223d8cff84aea778d629e36208 not found: ID does not exist" containerID="c8aceb7d7cb5478e055aed6659dad3dab6a9c2223d8cff84aea778d629e36208"
Nov 01 06:06:42.641791 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:42.641730    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8aceb7d7cb5478e055aed6659dad3dab6a9c2223d8cff84aea778d629e36208"} err="failed to get container status \"c8aceb7d7cb5478e055aed6659dad3dab6a9c2223d8cff84aea778d629e36208\": rpc error: code = NotFound desc = could not find container \"c8aceb7d7cb5478e055aed6659dad3dab6a9c2223d8cff84aea778d629e36208\": container with ID starting with c8aceb7d7cb5478e055aed6659dad3dab6a9c2223d8cff84aea778d629e36208 not found: ID does not exist"
Nov 01 06:06:42.799786 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:42.799745    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"]
Nov 01 06:06:42.850636 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:42.850602    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c5k96p4"]
Nov 01 06:06:42.949265 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:42.949170    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr" podUID="8bad5155-b5e5-4352-93be-5a3230c62cf1" containerName="payload-processing" containerID="cri-o://795290df61a6f139e95deddbd81912aeae36af30fb15226fb5326ebe8198e86c" gracePeriod=25
Nov 01 06:06:42.965586 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:42.965558    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c9e6d41-5c7c-4523-94fb-16d3b0bd263a" path="/var/lib/kubelet/pods/6c9e6d41-5c7c-4523-94fb-16d3b0bd263a/volumes"
Nov 01 06:06:43.063502 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.063457    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg" podUID="ab0d2dd6-34d8-45fa-953b-b70f15835c55" containerName="payload-pre-processing" containerID="cri-o://2ac52a79b6d01dd1df7b6d157bc6a8561ecdca7793d50f3a7ffdaf02f550f297" gracePeriod=25
Nov 01 06:06:43.090897 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.090869    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-8444n_1f121a97-b0e6-422d-ac1b-f3a235b333e9/payload-processing/0.log"
Nov 01 06:06:43.237406 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.237380    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr"
Nov 01 06:06:43.316109 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.316084    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-8444n_1f121a97-b0e6-422d-ac1b-f3a235b333e9/payload-processing/0.log"
Nov 01 06:06:43.359158 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.358855    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg"
Nov 01 06:06:43.401564 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.401501    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bad5155-b5e5-4352-93be-5a3230c62cf1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bad5155-b5e5-4352-93be-5a3230c62cf1-plugins-config-volume\") pod \"8bad5155-b5e5-4352-93be-5a3230c62cf1\" (UID: \"8bad5155-b5e5-4352-93be-5a3230c62cf1\") "
Nov 01 06:06:43.401780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.401672    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bad5155-b5e5-4352-93be-5a3230c62cf1-kube-api-access-42pqr\" (UniqueName: \"kubernetes.io/projected/8bad5155-b5e5-4352-93be-5a3230c62cf1-kube-api-access-42pqr\") pod \"8bad5155-b5e5-4352-93be-5a3230c62cf1\" (UID: \"8bad5155-b5e5-4352-93be-5a3230c62cf1\") "
Nov 01 06:06:43.401965 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.401912    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bad5155-b5e5-4352-93be-5a3230c62cf1-plugins-config-volume" pod "8bad5155-b5e5-4352-93be-5a3230c62cf1" (UID: "8bad5155-b5e5-4352-93be-5a3230c62cf1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:06:43.402457 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.402432    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bad5155-b5e5-4352-93be-5a3230c62cf1-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:06:43.404123 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.404062    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bad5155-b5e5-4352-93be-5a3230c62cf1-kube-api-access-42pqr" pod "8bad5155-b5e5-4352-93be-5a3230c62cf1" (UID: "8bad5155-b5e5-4352-93be-5a3230c62cf1"). InnerVolumeSpecName "kube-api-access-42pqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:06:43.503794 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.503700    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0d2dd6-34d8-45fa-953b-b70f15835c55-kube-api-access-5mlds\" (UniqueName: \"kubernetes.io/projected/ab0d2dd6-34d8-45fa-953b-b70f15835c55-kube-api-access-5mlds\") pod \"ab0d2dd6-34d8-45fa-953b-b70f15835c55\" (UID: \"ab0d2dd6-34d8-45fa-953b-b70f15835c55\") "
Nov 01 06:06:43.503973 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.503880    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab0d2dd6-34d8-45fa-953b-b70f15835c55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab0d2dd6-34d8-45fa-953b-b70f15835c55-plugins-config-volume\") pod \"ab0d2dd6-34d8-45fa-953b-b70f15835c55\" (UID: \"ab0d2dd6-34d8-45fa-953b-b70f15835c55\") "
Nov 01 06:06:43.504280 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.504238    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab0d2dd6-34d8-45fa-953b-b70f15835c55-plugins-config-volume" pod "ab0d2dd6-34d8-45fa-953b-b70f15835c55" (UID: "ab0d2dd6-34d8-45fa-953b-b70f15835c55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:06:43.504491 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.504474    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab0d2dd6-34d8-45fa-953b-b70f15835c55-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:06:43.504491 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.504491    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42pqr\" (UniqueName: \"kubernetes.io/projected/8bad5155-b5e5-4352-93be-5a3230c62cf1-kube-api-access-42pqr\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:06:43.505776 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.505754    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0d2dd6-34d8-45fa-953b-b70f15835c55-kube-api-access-5mlds" pod "ab0d2dd6-34d8-45fa-953b-b70f15835c55" (UID: "ab0d2dd6-34d8-45fa-953b-b70f15835c55"). InnerVolumeSpecName "kube-api-access-5mlds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:06:43.606236 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.606202    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mlds\" (UniqueName: \"kubernetes.io/projected/ab0d2dd6-34d8-45fa-953b-b70f15835c55-kube-api-access-5mlds\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:06:43.638361 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.638308    5619 generic.go:356] "Generic (PLEG): container finished" podID="8bad5155-b5e5-4352-93be-5a3230c62cf1" containerID="795290df61a6f139e95deddbd81912aeae36af30fb15226fb5326ebe8198e86c" exitCode=0
Nov 01 06:06:43.638817 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.638390    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr"
Nov 01 06:06:43.638817 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.638393    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr" event={"ID":"8bad5155-b5e5-4352-93be-5a3230c62cf1","Type":"ContainerDied","Data":"795290df61a6f139e95deddbd81912aeae36af30fb15226fb5326ebe8198e86c"}
Nov 01 06:06:43.638817 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.638473    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr" event={"ID":"8bad5155-b5e5-4352-93be-5a3230c62cf1","Type":"ContainerDied","Data":"78f1d036f8195eb8733d79c4406a2da166570ff90fdb3108879e6b472703e618"}
Nov 01 06:06:43.638817 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.638508    5619 scope.go:122] "RemoveContainer" containerID="795290df61a6f139e95deddbd81912aeae36af30fb15226fb5326ebe8198e86c"
Nov 01 06:06:43.641664 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.641635    5619 generic.go:356] "Generic (PLEG): container finished" podID="ab0d2dd6-34d8-45fa-953b-b70f15835c55" containerID="2ac52a79b6d01dd1df7b6d157bc6a8561ecdca7793d50f3a7ffdaf02f550f297" exitCode=0
Nov 01 06:06:43.641805 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.641678    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg"
Nov 01 06:06:43.641805 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.641715    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg" event={"ID":"ab0d2dd6-34d8-45fa-953b-b70f15835c55","Type":"ContainerDied","Data":"2ac52a79b6d01dd1df7b6d157bc6a8561ecdca7793d50f3a7ffdaf02f550f297"}
Nov 01 06:06:43.641805 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.641773    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg" event={"ID":"ab0d2dd6-34d8-45fa-953b-b70f15835c55","Type":"ContainerDied","Data":"1066abc94c9d3aaab0091757488061c14a328344bbaec9c41339fa7bc664aea7"}
Nov 01 06:06:43.649841 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.649814    5619 scope.go:122] "RemoveContainer" containerID="795290df61a6f139e95deddbd81912aeae36af30fb15226fb5326ebe8198e86c"
Nov 01 06:06:43.650223 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:06:43.650196    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795290df61a6f139e95deddbd81912aeae36af30fb15226fb5326ebe8198e86c\": container with ID starting with 795290df61a6f139e95deddbd81912aeae36af30fb15226fb5326ebe8198e86c not found: ID does not exist" containerID="795290df61a6f139e95deddbd81912aeae36af30fb15226fb5326ebe8198e86c"
Nov 01 06:06:43.650386 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.650234    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795290df61a6f139e95deddbd81912aeae36af30fb15226fb5326ebe8198e86c"} err="failed to get container status \"795290df61a6f139e95deddbd81912aeae36af30fb15226fb5326ebe8198e86c\": rpc error: code = NotFound desc = could not find container \"795290df61a6f139e95deddbd81912aeae36af30fb15226fb5326ebe8198e86c\": container with ID starting with 795290df61a6f139e95deddbd81912aeae36af30fb15226fb5326ebe8198e86c not found: ID does not exist"
Nov 01 06:06:43.650386 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.650258    5619 scope.go:122] "RemoveContainer" containerID="2ac52a79b6d01dd1df7b6d157bc6a8561ecdca7793d50f3a7ffdaf02f550f297"
Nov 01 06:06:43.660363 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.660316    5619 scope.go:122] "RemoveContainer" containerID="2ac52a79b6d01dd1df7b6d157bc6a8561ecdca7793d50f3a7ffdaf02f550f297"
Nov 01 06:06:43.660640 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:06:43.660619    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ac52a79b6d01dd1df7b6d157bc6a8561ecdca7793d50f3a7ffdaf02f550f297\": container with ID starting with 2ac52a79b6d01dd1df7b6d157bc6a8561ecdca7793d50f3a7ffdaf02f550f297 not found: ID does not exist" containerID="2ac52a79b6d01dd1df7b6d157bc6a8561ecdca7793d50f3a7ffdaf02f550f297"
Nov 01 06:06:43.660696 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.660648    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ac52a79b6d01dd1df7b6d157bc6a8561ecdca7793d50f3a7ffdaf02f550f297"} err="failed to get container status \"2ac52a79b6d01dd1df7b6d157bc6a8561ecdca7793d50f3a7ffdaf02f550f297\": rpc error: code = NotFound desc = could not find container \"2ac52a79b6d01dd1df7b6d157bc6a8561ecdca7793d50f3a7ffdaf02f550f297\": container with ID starting with 2ac52a79b6d01dd1df7b6d157bc6a8561ecdca7793d50f3a7ffdaf02f550f297 not found: ID does not exist"
Nov 01 06:06:43.806839 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.806745    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr"]
Nov 01 06:06:43.811094 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.811059    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d3f8d-797f76dc85-h9pjr"]
Nov 01 06:06:43.927411 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.927375    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg"]
Nov 01 06:06:43.932208 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.932180    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-57fd99c9b7-z54jg"]
Nov 01 06:06:43.974163 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:43.974128    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:44.113239 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:06:44.113203    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-cfe368/agc-cred-83f5507c1f28: secret "agc-cred-83f5507c1f28" not found
Nov 01 06:06:44.113239 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:06:44.113239    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p: secret "agc-cred-83f5507c1f28" not found
Nov 01 06:06:44.113558 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:06:44.113316    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23b253a4-3245-4e42-a164-626dd69baf75-provider-credentials podName:23b253a4-3245-4e42-a164-626dd69baf75 nodeName:}" failed. No retries permitted until 2026-11-01 06:06:48.113295537 +0000 UTC m=+1245.324119037 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/23b253a4-3245-4e42-a164-626dd69baf75-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p" (UID: "23b253a4-3245-4e42-a164-626dd69baf75") : secret "agc-cred-83f5507c1f28" not found
Nov 01 06:06:44.962728 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:44.962698    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bad5155-b5e5-4352-93be-5a3230c62cf1" path="/var/lib/kubelet/pods/8bad5155-b5e5-4352-93be-5a3230c62cf1/volumes"
Nov 01 06:06:44.963109 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:44.963078    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab0d2dd6-34d8-45fa-953b-b70f15835c55" path="/var/lib/kubelet/pods/ab0d2dd6-34d8-45fa-953b-b70f15835c55/volumes"
Nov 01 06:06:45.618429 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:45.618373    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p" podUID="23b253a4-3245-4e42-a164-626dd69baf75" containerName="payload-processing" containerID="cri-o://5f262be1ef193970bc9e1d46540554fba2e48d9a5d64bc3f5fbfab52af02a873" gracePeriod=25
Nov 01 06:06:45.905233 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:45.905208    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"
Nov 01 06:06:46.037283 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:46.037249    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23b253a4-3245-4e42-a164-626dd69baf75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b253a4-3245-4e42-a164-626dd69baf75-plugins-config-volume\") pod \"23b253a4-3245-4e42-a164-626dd69baf75\" (UID: \"23b253a4-3245-4e42-a164-626dd69baf75\") "
Nov 01 06:06:46.037805 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:46.037308    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23b253a4-3245-4e42-a164-626dd69baf75-kube-api-access-2bhn9\" (UniqueName: \"kubernetes.io/projected/23b253a4-3245-4e42-a164-626dd69baf75-kube-api-access-2bhn9\") pod \"23b253a4-3245-4e42-a164-626dd69baf75\" (UID: \"23b253a4-3245-4e42-a164-626dd69baf75\") "
Nov 01 06:06:46.037805 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:46.037498    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23b253a4-3245-4e42-a164-626dd69baf75-provider-credentials\" (UniqueName: \"kubernetes.io/projected/23b253a4-3245-4e42-a164-626dd69baf75-provider-credentials\") pod \"23b253a4-3245-4e42-a164-626dd69baf75\" (UID: \"23b253a4-3245-4e42-a164-626dd69baf75\") "
Nov 01 06:06:46.037805 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:46.037543    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23b253a4-3245-4e42-a164-626dd69baf75-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23b253a4-3245-4e42-a164-626dd69baf75-routing-overlay-volume\") pod \"23b253a4-3245-4e42-a164-626dd69baf75\" (UID: \"23b253a4-3245-4e42-a164-626dd69baf75\") "
Nov 01 06:06:46.037805 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:46.037676    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b253a4-3245-4e42-a164-626dd69baf75-plugins-config-volume" pod "23b253a4-3245-4e42-a164-626dd69baf75" (UID: "23b253a4-3245-4e42-a164-626dd69baf75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:06:46.038209 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:46.038182    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b253a4-3245-4e42-a164-626dd69baf75-routing-overlay-volume" pod "23b253a4-3245-4e42-a164-626dd69baf75" (UID: "23b253a4-3245-4e42-a164-626dd69baf75"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:06:46.039006 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:46.038981    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b253a4-3245-4e42-a164-626dd69baf75-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:06:46.039268 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:46.039245    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23b253a4-3245-4e42-a164-626dd69baf75-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:06:46.039978 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:46.039954    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b253a4-3245-4e42-a164-626dd69baf75-provider-credentials" pod "23b253a4-3245-4e42-a164-626dd69baf75" (UID: "23b253a4-3245-4e42-a164-626dd69baf75"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:06:46.040066 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:46.040011    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b253a4-3245-4e42-a164-626dd69baf75-kube-api-access-2bhn9" pod "23b253a4-3245-4e42-a164-626dd69baf75" (UID: "23b253a4-3245-4e42-a164-626dd69baf75"). InnerVolumeSpecName "kube-api-access-2bhn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:06:46.141231 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:46.141126    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bhn9\" (UniqueName: \"kubernetes.io/projected/23b253a4-3245-4e42-a164-626dd69baf75-kube-api-access-2bhn9\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:06:46.141231 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:46.141172    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23b253a4-3245-4e42-a164-626dd69baf75-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:06:46.663342 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:46.663290    5619 generic.go:356] "Generic (PLEG): container finished" podID="23b253a4-3245-4e42-a164-626dd69baf75" containerID="5f262be1ef193970bc9e1d46540554fba2e48d9a5d64bc3f5fbfab52af02a873" exitCode=0
Nov 01 06:06:46.663550 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:46.663367    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"
Nov 01 06:06:46.663550 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:46.663359    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p" event={"ID":"23b253a4-3245-4e42-a164-626dd69baf75","Type":"ContainerDied","Data":"5f262be1ef193970bc9e1d46540554fba2e48d9a5d64bc3f5fbfab52af02a873"}
Nov 01 06:06:46.663550 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:46.663502    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p" event={"ID":"23b253a4-3245-4e42-a164-626dd69baf75","Type":"ContainerDied","Data":"9d8216ccf5cafbbec5a7c91a24768183f198dbf79dedcaf99979bd6db273412f"}
Nov 01 06:06:46.663550 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:46.663532    5619 scope.go:122] "RemoveContainer" containerID="5f262be1ef193970bc9e1d46540554fba2e48d9a5d64bc3f5fbfab52af02a873"
Nov 01 06:06:46.673075 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:46.673057    5619 scope.go:122] "RemoveContainer" containerID="5f262be1ef193970bc9e1d46540554fba2e48d9a5d64bc3f5fbfab52af02a873"
Nov 01 06:06:46.673398 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:06:46.673369    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f262be1ef193970bc9e1d46540554fba2e48d9a5d64bc3f5fbfab52af02a873\": container with ID starting with 5f262be1ef193970bc9e1d46540554fba2e48d9a5d64bc3f5fbfab52af02a873 not found: ID does not exist" containerID="5f262be1ef193970bc9e1d46540554fba2e48d9a5d64bc3f5fbfab52af02a873"
Nov 01 06:06:46.673467 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:46.673411    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f262be1ef193970bc9e1d46540554fba2e48d9a5d64bc3f5fbfab52af02a873"} err="failed to get container status \"5f262be1ef193970bc9e1d46540554fba2e48d9a5d64bc3f5fbfab52af02a873\": rpc error: code = NotFound desc = could not find container \"5f262be1ef193970bc9e1d46540554fba2e48d9a5d64bc3f5fbfab52af02a873\": container with ID starting with 5f262be1ef193970bc9e1d46540554fba2e48d9a5d64bc3f5fbfab52af02a873 not found: ID does not exist"
Nov 01 06:06:46.831286 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:46.831249    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"]
Nov 01 06:06:46.883442 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:46.883411    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-7947dm69p"]
Nov 01 06:06:46.963388 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:46.963264    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23b253a4-3245-4e42-a164-626dd69baf75" path="/var/lib/kubelet/pods/23b253a4-3245-4e42-a164-626dd69baf75/volumes"
Nov 01 06:06:48.763852 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:48.763818    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:52.802261 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:52.778668    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:53.240301 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:53.240206    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:53.617136 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:53.617100    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:55.879874 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:06:55.879842    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:06:58.400737 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:58.400694    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:06:59.315225 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:06:59.315186    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:00.008008 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:00.007971    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns"]
Nov 01 06:07:00.369206 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:00.369169    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:00.515271 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:00.515236    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:03.666514 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:03.666479    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:05.008984 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:05.008923    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns" podUID="97b887a3-3119-4b1a-a642-9a5c7c21ff5a" containerName="payload-processing" containerID="cri-o://febabc79ba6c36db7ac1d2c0bea7045c9f9c8ed0be52353c2d278bd9c950ac99" gracePeriod=25
Nov 01 06:07:05.144408 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:05.144375    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:08.438345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:08.436690    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns" podUID="97b887a3-3119-4b1a-a642-9a5c7c21ff5a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:07:09.647258 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:09.647212    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"]
Nov 01 06:07:09.647671 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:09.647512    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg" podUID="f5a8793d-01dc-42a1-83bd-37f944265ce1" containerName="maas-api" containerID="cri-o://e6f77c99ca64e1d8cf87fac2cea11a33fca69c30a62b28426358e6c4f2adee91" gracePeriod=30
Nov 01 06:07:09.747995 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:09.747960    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:09.866718 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:09.866403    5619 generic.go:356] "Generic (PLEG): container finished" podID="f5a8793d-01dc-42a1-83bd-37f944265ce1" containerID="e6f77c99ca64e1d8cf87fac2cea11a33fca69c30a62b28426358e6c4f2adee91" exitCode=0
Nov 01 06:07:09.866718 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:09.866598    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg" event={"ID":"f5a8793d-01dc-42a1-83bd-37f944265ce1","Type":"ContainerDied","Data":"e6f77c99ca64e1d8cf87fac2cea11a33fca69c30a62b28426358e6c4f2adee91"}
Nov 01 06:07:09.943676 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:09.943643    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"
Nov 01 06:07:10.063347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:10.054659    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5a8793d-01dc-42a1-83bd-37f944265ce1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a8793d-01dc-42a1-83bd-37f944265ce1-openshift-service-ca\") pod \"f5a8793d-01dc-42a1-83bd-37f944265ce1\" (UID: \"f5a8793d-01dc-42a1-83bd-37f944265ce1\") "
Nov 01 06:07:10.063347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:10.054760    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5a8793d-01dc-42a1-83bd-37f944265ce1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5a8793d-01dc-42a1-83bd-37f944265ce1-metrics-certs\") pod \"f5a8793d-01dc-42a1-83bd-37f944265ce1\" (UID: \"f5a8793d-01dc-42a1-83bd-37f944265ce1\") "
Nov 01 06:07:10.063347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:10.054878    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5a8793d-01dc-42a1-83bd-37f944265ce1-kube-api-access-hlwz9\" (UniqueName: \"kubernetes.io/projected/f5a8793d-01dc-42a1-83bd-37f944265ce1-kube-api-access-hlwz9\") pod \"f5a8793d-01dc-42a1-83bd-37f944265ce1\" (UID: \"f5a8793d-01dc-42a1-83bd-37f944265ce1\") "
Nov 01 06:07:10.063347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:10.054923    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5a8793d-01dc-42a1-83bd-37f944265ce1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5a8793d-01dc-42a1-83bd-37f944265ce1-maas-api-tls\") pod \"f5a8793d-01dc-42a1-83bd-37f944265ce1\" (UID: \"f5a8793d-01dc-42a1-83bd-37f944265ce1\") "
Nov 01 06:07:10.073612 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:10.072834    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5a8793d-01dc-42a1-83bd-37f944265ce1-openshift-service-ca" pod "f5a8793d-01dc-42a1-83bd-37f944265ce1" (UID: "f5a8793d-01dc-42a1-83bd-37f944265ce1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:07:10.079583 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:10.079507    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5a8793d-01dc-42a1-83bd-37f944265ce1-metrics-certs" pod "f5a8793d-01dc-42a1-83bd-37f944265ce1" (UID: "f5a8793d-01dc-42a1-83bd-37f944265ce1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:07:10.095183 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:10.085455    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5a8793d-01dc-42a1-83bd-37f944265ce1-maas-api-tls" pod "f5a8793d-01dc-42a1-83bd-37f944265ce1" (UID: "f5a8793d-01dc-42a1-83bd-37f944265ce1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:07:10.112346 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:10.095744    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a8793d-01dc-42a1-83bd-37f944265ce1-kube-api-access-hlwz9" pod "f5a8793d-01dc-42a1-83bd-37f944265ce1" (UID: "f5a8793d-01dc-42a1-83bd-37f944265ce1"). InnerVolumeSpecName "kube-api-access-hlwz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:07:10.156797 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:10.156757    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlwz9\" (UniqueName: \"kubernetes.io/projected/f5a8793d-01dc-42a1-83bd-37f944265ce1-kube-api-access-hlwz9\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:10.156955 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:10.156802    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5a8793d-01dc-42a1-83bd-37f944265ce1-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:10.156955 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:10.156849    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a8793d-01dc-42a1-83bd-37f944265ce1-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:10.156955 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:10.156864    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5a8793d-01dc-42a1-83bd-37f944265ce1-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:10.876019 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:10.875924    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg" event={"ID":"f5a8793d-01dc-42a1-83bd-37f944265ce1","Type":"ContainerDied","Data":"ef84abec390eba129a7b3c3e43c73d64a9d7568e19d22d2e425d3f20a1fd3bb6"}
Nov 01 06:07:10.876019 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:10.875998    5619 scope.go:122] "RemoveContainer" containerID="e6f77c99ca64e1d8cf87fac2cea11a33fca69c30a62b28426358e6c4f2adee91"
Nov 01 06:07:10.876592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:10.876150    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"
Nov 01 06:07:11.023632 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:11.023598    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:11.049934 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:11.049907    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"]
Nov 01 06:07:11.055249 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:11.055217    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d2807-646b7847ff-k5xjg"]
Nov 01 06:07:11.180167 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:11.180083    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:11.337681 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:11.337646    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:11.609185 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:11.609153    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:12.422949 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:12.422915    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:12.792280 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:12.792199    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:12.962791 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:12.962761    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5a8793d-01dc-42a1-83bd-37f944265ce1" path="/var/lib/kubelet/pods/f5a8793d-01dc-42a1-83bd-37f944265ce1/volumes"
Nov 01 06:07:13.396291 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:13.396257    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:13.401885 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:13.401849    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"]
Nov 01 06:07:13.402069 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:13.402047    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k" podUID="680af3cd-644f-459b-81bf-686fdc62b198" containerName="istio-proxy" containerID="cri-o://2fcfb1f692f66039c3596e1a41ca1775222c998663b4b8eabd19212a81215f09" gracePeriod=30
Nov 01 06:07:14.223984 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:14.223949    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:16.931478 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:16.931439    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:18.403855 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.403815    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns" podUID="97b887a3-3119-4b1a-a642-9a5c7c21ff5a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:07:18.430156 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.430126    5619 patch_prober.go:29] interesting pod/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 01 06:07:18.430402 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.430371    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k" podUID="680af3cd-644f-459b-81bf-686fdc62b198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 01 06:07:18.481145 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.481111    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:18.679298 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.679273    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 06:07:18.780646 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.780596    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/680af3cd-644f-459b-81bf-686fdc62b198-istio-token\" (UniqueName: \"kubernetes.io/projected/680af3cd-644f-459b-81bf-686fdc62b198-istio-token\") pod \"680af3cd-644f-459b-81bf-686fdc62b198\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") "
Nov 01 06:07:18.780646 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.780633    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/680af3cd-644f-459b-81bf-686fdc62b198-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/680af3cd-644f-459b-81bf-686fdc62b198-istio-podinfo\") pod \"680af3cd-644f-459b-81bf-686fdc62b198\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") "
Nov 01 06:07:18.780859 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.780726    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-istio-data\") pod \"680af3cd-644f-459b-81bf-686fdc62b198\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") "
Nov 01 06:07:18.780859 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.780790    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-workload-socket\") pod \"680af3cd-644f-459b-81bf-686fdc62b198\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") "
Nov 01 06:07:18.780859 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.780832    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/680af3cd-644f-459b-81bf-686fdc62b198-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/680af3cd-644f-459b-81bf-686fdc62b198-istiod-ca-cert\") pod \"680af3cd-644f-459b-81bf-686fdc62b198\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") "
Nov 01 06:07:18.781014 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.780913    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/680af3cd-644f-459b-81bf-686fdc62b198-kube-api-access-k29ln\" (UniqueName: \"kubernetes.io/projected/680af3cd-644f-459b-81bf-686fdc62b198-kube-api-access-k29ln\") pod \"680af3cd-644f-459b-81bf-686fdc62b198\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") "
Nov 01 06:07:18.781014 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.780965    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-workload-certs\") pod \"680af3cd-644f-459b-81bf-686fdc62b198\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") "
Nov 01 06:07:18.781131 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.781013    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-istio-data" pod "680af3cd-644f-459b-81bf-686fdc62b198" (UID: "680af3cd-644f-459b-81bf-686fdc62b198"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:07:18.781131 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.781061    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-istio-envoy\") pod \"680af3cd-644f-459b-81bf-686fdc62b198\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") "
Nov 01 06:07:18.781131 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.781062    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-workload-socket" pod "680af3cd-644f-459b-81bf-686fdc62b198" (UID: "680af3cd-644f-459b-81bf-686fdc62b198"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:07:18.781131 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.781094    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-credential-socket\") pod \"680af3cd-644f-459b-81bf-686fdc62b198\" (UID: \"680af3cd-644f-459b-81bf-686fdc62b198\") "
Nov 01 06:07:18.781625 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.781544    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-credential-socket" pod "680af3cd-644f-459b-81bf-686fdc62b198" (UID: "680af3cd-644f-459b-81bf-686fdc62b198"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:07:18.781625 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.781553    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:18.781625 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.781572    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-workload-certs" pod "680af3cd-644f-459b-81bf-686fdc62b198" (UID: "680af3cd-644f-459b-81bf-686fdc62b198"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:07:18.781625 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.781599    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:18.781977 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.781530    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/680af3cd-644f-459b-81bf-686fdc62b198-istiod-ca-cert" pod "680af3cd-644f-459b-81bf-686fdc62b198" (UID: "680af3cd-644f-459b-81bf-686fdc62b198"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:07:18.783102 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.783060    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/680af3cd-644f-459b-81bf-686fdc62b198-istio-podinfo" pod "680af3cd-644f-459b-81bf-686fdc62b198" (UID: "680af3cd-644f-459b-81bf-686fdc62b198"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:07:18.783253 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.783234    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680af3cd-644f-459b-81bf-686fdc62b198-istio-token" pod "680af3cd-644f-459b-81bf-686fdc62b198" (UID: "680af3cd-644f-459b-81bf-686fdc62b198"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:07:18.783342 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.783238    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-istio-envoy" pod "680af3cd-644f-459b-81bf-686fdc62b198" (UID: "680af3cd-644f-459b-81bf-686fdc62b198"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:07:18.783342 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.783272    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680af3cd-644f-459b-81bf-686fdc62b198-kube-api-access-k29ln" pod "680af3cd-644f-459b-81bf-686fdc62b198" (UID: "680af3cd-644f-459b-81bf-686fdc62b198"). InnerVolumeSpecName "kube-api-access-k29ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:07:18.882898 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.882869    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/680af3cd-644f-459b-81bf-686fdc62b198-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:18.882898 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.882897    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k29ln\" (UniqueName: \"kubernetes.io/projected/680af3cd-644f-459b-81bf-686fdc62b198-kube-api-access-k29ln\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:18.883089 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.882907    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:18.883089 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.882937    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:18.883089 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.882947    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/680af3cd-644f-459b-81bf-686fdc62b198-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:18.883089 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.882956    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/680af3cd-644f-459b-81bf-686fdc62b198-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:18.883089 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.882964    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/680af3cd-644f-459b-81bf-686fdc62b198-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:18.927077 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.926985    5619 generic.go:356] "Generic (PLEG): container finished" podID="680af3cd-644f-459b-81bf-686fdc62b198" containerID="2fcfb1f692f66039c3596e1a41ca1775222c998663b4b8eabd19212a81215f09" exitCode=0
Nov 01 06:07:18.927077 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.927033    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"
Nov 01 06:07:18.927248 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.927070    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k" event={"ID":"680af3cd-644f-459b-81bf-686fdc62b198","Type":"ContainerDied","Data":"2fcfb1f692f66039c3596e1a41ca1775222c998663b4b8eabd19212a81215f09"}
Nov 01 06:07:18.927248 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.927151    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k" event={"ID":"680af3cd-644f-459b-81bf-686fdc62b198","Type":"ContainerDied","Data":"34e31a808679d69ceb5cc0a32d9b60f3d99a27e5c8e1fefe293d1d37d2fca6dd"}
Nov 01 06:07:18.927248 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.927173    5619 scope.go:122] "RemoveContainer" containerID="2fcfb1f692f66039c3596e1a41ca1775222c998663b4b8eabd19212a81215f09"
Nov 01 06:07:18.937750 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.937732    5619 scope.go:122] "RemoveContainer" containerID="57f27d5a1a6076a06b7ec3c7a9a746cd1b66533d09968679d6c85920a711be38"
Nov 01 06:07:18.946437 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.946418    5619 scope.go:122] "RemoveContainer" containerID="2fcfb1f692f66039c3596e1a41ca1775222c998663b4b8eabd19212a81215f09"
Nov 01 06:07:18.946690 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:07:18.946670    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fcfb1f692f66039c3596e1a41ca1775222c998663b4b8eabd19212a81215f09\": container with ID starting with 2fcfb1f692f66039c3596e1a41ca1775222c998663b4b8eabd19212a81215f09 not found: ID does not exist" containerID="2fcfb1f692f66039c3596e1a41ca1775222c998663b4b8eabd19212a81215f09"
Nov 01 06:07:18.946768 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.946703    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fcfb1f692f66039c3596e1a41ca1775222c998663b4b8eabd19212a81215f09"} err="failed to get container status \"2fcfb1f692f66039c3596e1a41ca1775222c998663b4b8eabd19212a81215f09\": rpc error: code = NotFound desc = could not find container \"2fcfb1f692f66039c3596e1a41ca1775222c998663b4b8eabd19212a81215f09\": container with ID starting with 2fcfb1f692f66039c3596e1a41ca1775222c998663b4b8eabd19212a81215f09 not found: ID does not exist"
Nov 01 06:07:18.946768 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.946728    5619 scope.go:122] "RemoveContainer" containerID="57f27d5a1a6076a06b7ec3c7a9a746cd1b66533d09968679d6c85920a711be38"
Nov 01 06:07:18.946979 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:07:18.946960    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57f27d5a1a6076a06b7ec3c7a9a746cd1b66533d09968679d6c85920a711be38\": container with ID starting with 57f27d5a1a6076a06b7ec3c7a9a746cd1b66533d09968679d6c85920a711be38 not found: ID does not exist" containerID="57f27d5a1a6076a06b7ec3c7a9a746cd1b66533d09968679d6c85920a711be38"
Nov 01 06:07:18.947021 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:18.946986    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f27d5a1a6076a06b7ec3c7a9a746cd1b66533d09968679d6c85920a711be38"} err="failed to get container status \"57f27d5a1a6076a06b7ec3c7a9a746cd1b66533d09968679d6c85920a711be38\": rpc error: code = NotFound desc = could not find container \"57f27d5a1a6076a06b7ec3c7a9a746cd1b66533d09968679d6c85920a711be38\": container with ID starting with 57f27d5a1a6076a06b7ec3c7a9a746cd1b66533d09968679d6c85920a711be38 not found: ID does not exist"
Nov 01 06:07:19.095668 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:19.095632    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"]
Nov 01 06:07:19.102004 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:19.101969    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0d2807-openshift-default-54f678c8f-r9w8k"]
Nov 01 06:07:20.139082 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:20.139038    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:20.330985 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:20.330949    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:20.473122 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:20.473031    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:20.612987 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:20.612949    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:20.948562 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:20.948526    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:20.963264 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:20.963232    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="680af3cd-644f-459b-81bf-686fdc62b198" path="/var/lib/kubelet/pods/680af3cd-644f-459b-81bf-686fdc62b198/volumes"
Nov 01 06:07:22.063747 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.063712    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq"]
Nov 01 06:07:22.064544 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.064524    5619 cpu_manager.go:414] "containerMap: removing container" podUID="ab0d2dd6-34d8-45fa-953b-b70f15835c55" containerName="payload-pre-processing"
Nov 01 06:07:22.064592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.064546    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0d2dd6-34d8-45fa-953b-b70f15835c55" containerName="payload-pre-processing"
Nov 01 06:07:22.064592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.064563    5619 cpu_manager.go:414] "containerMap: removing container" podUID="8bad5155-b5e5-4352-93be-5a3230c62cf1" containerName="payload-processing"
Nov 01 06:07:22.064592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.064573    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bad5155-b5e5-4352-93be-5a3230c62cf1" containerName="payload-processing"
Nov 01 06:07:22.064713 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.064594    5619 cpu_manager.go:414] "containerMap: removing container" podUID="680af3cd-644f-459b-81bf-686fdc62b198" containerName="istio-proxy"
Nov 01 06:07:22.064713 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.064601    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680af3cd-644f-459b-81bf-686fdc62b198" containerName="istio-proxy"
Nov 01 06:07:22.064713 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.064636    5619 cpu_manager.go:414] "containerMap: removing container" podUID="f5a8793d-01dc-42a1-83bd-37f944265ce1" containerName="maas-api"
Nov 01 06:07:22.064713 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.064643    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5a8793d-01dc-42a1-83bd-37f944265ce1" containerName="maas-api"
Nov 01 06:07:22.064713 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.064663    5619 cpu_manager.go:414] "containerMap: removing container" podUID="23b253a4-3245-4e42-a164-626dd69baf75" containerName="payload-processing"
Nov 01 06:07:22.064713 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.064670    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23b253a4-3245-4e42-a164-626dd69baf75" containerName="payload-processing"
Nov 01 06:07:22.064713 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.064684    5619 cpu_manager.go:414] "containerMap: removing container" podUID="6c9e6d41-5c7c-4523-94fb-16d3b0bd263a" containerName="payload-processing"
Nov 01 06:07:22.064713 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.064692    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c9e6d41-5c7c-4523-94fb-16d3b0bd263a" containerName="payload-processing"
Nov 01 06:07:22.064949 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.064802    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="23b253a4-3245-4e42-a164-626dd69baf75" containerName="payload-processing"
Nov 01 06:07:22.064949 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.064815    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c9e6d41-5c7c-4523-94fb-16d3b0bd263a" containerName="payload-processing"
Nov 01 06:07:22.064949 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.064825    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="680af3cd-644f-459b-81bf-686fdc62b198" containerName="istio-proxy"
Nov 01 06:07:22.064949 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.064836    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0d2dd6-34d8-45fa-953b-b70f15835c55" containerName="payload-pre-processing"
Nov 01 06:07:22.064949 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.064845    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bad5155-b5e5-4352-93be-5a3230c62cf1" containerName="payload-processing"
Nov 01 06:07:22.064949 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.064860    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5a8793d-01dc-42a1-83bd-37f944265ce1" containerName="maas-api"
Nov 01 06:07:22.070153 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.070129    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq"
Nov 01 06:07:22.148217 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.148079    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq"]
Nov 01 06:07:22.162285 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.162225    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qjkd\" (UniqueName: \"kubernetes.io/projected/8a586fea-87b6-4599-beed-36aba543e3c9-kube-api-access-8qjkd\") pod \"payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq\" (UID: \"8a586fea-87b6-4599-beed-36aba543e3c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq"
Nov 01 06:07:22.162700 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.162673    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a586fea-87b6-4599-beed-36aba543e3c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq\" (UID: \"8a586fea-87b6-4599-beed-36aba543e3c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq"
Nov 01 06:07:22.173927 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.173905    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2d3f8d\""
Nov 01 06:07:22.174417 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.174384    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2d3f8d-dockercfg-pxf4b\""
Nov 01 06:07:22.264034 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.263993    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qjkd\" (UniqueName: \"kubernetes.io/projected/8a586fea-87b6-4599-beed-36aba543e3c9-kube-api-access-8qjkd\") pod \"payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq\" (UID: \"8a586fea-87b6-4599-beed-36aba543e3c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq"
Nov 01 06:07:22.264239 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.264051    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a586fea-87b6-4599-beed-36aba543e3c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq\" (UID: \"8a586fea-87b6-4599-beed-36aba543e3c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq"
Nov 01 06:07:22.265018 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.264991    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a586fea-87b6-4599-beed-36aba543e3c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq\" (UID: \"8a586fea-87b6-4599-beed-36aba543e3c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq"
Nov 01 06:07:22.336682 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.336610    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qjkd\" (UniqueName: \"kubernetes.io/projected/8a586fea-87b6-4599-beed-36aba543e3c9-kube-api-access-8qjkd\") pod \"payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq\" (UID: \"8a586fea-87b6-4599-beed-36aba543e3c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq"
Nov 01 06:07:22.384481 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.384449    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq"
Nov 01 06:07:22.826132 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:22.826089    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq"]
Nov 01 06:07:22.830611 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:07:22.830583    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a586fea_87b6_4599_beed_36aba543e3c9.slice/crio-5922c421e8902e316db3e27d96515a5fec981432315295ee01d9c248fce0f659 WatchSource:0}: Error finding container 5922c421e8902e316db3e27d96515a5fec981432315295ee01d9c248fce0f659: Status 404 returned error can't find the container with id 5922c421e8902e316db3e27d96515a5fec981432315295ee01d9c248fce0f659
Nov 01 06:07:23.008333 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:23.008285    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq" event={"ID":"8a586fea-87b6-4599-beed-36aba543e3c9","Type":"ContainerStarted","Data":"5922c421e8902e316db3e27d96515a5fec981432315295ee01d9c248fce0f659"}
Nov 01 06:07:24.120868 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:24.120816    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:25.340336 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:25.340298    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns"
Nov 01 06:07:25.392012 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:25.390852    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97b887a3-3119-4b1a-a642-9a5c7c21ff5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b887a3-3119-4b1a-a642-9a5c7c21ff5a-plugins-config-volume\") pod \"97b887a3-3119-4b1a-a642-9a5c7c21ff5a\" (UID: \"97b887a3-3119-4b1a-a642-9a5c7c21ff5a\") "
Nov 01 06:07:25.392012 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:25.391085    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b887a3-3119-4b1a-a642-9a5c7c21ff5a-kube-api-access-lkhfz\" (UniqueName: \"kubernetes.io/projected/97b887a3-3119-4b1a-a642-9a5c7c21ff5a-kube-api-access-lkhfz\") pod \"97b887a3-3119-4b1a-a642-9a5c7c21ff5a\" (UID: \"97b887a3-3119-4b1a-a642-9a5c7c21ff5a\") "
Nov 01 06:07:25.393303 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:25.393241    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b887a3-3119-4b1a-a642-9a5c7c21ff5a-plugins-config-volume" pod "97b887a3-3119-4b1a-a642-9a5c7c21ff5a" (UID: "97b887a3-3119-4b1a-a642-9a5c7c21ff5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:07:25.405817 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:25.405773    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b887a3-3119-4b1a-a642-9a5c7c21ff5a-kube-api-access-lkhfz" pod "97b887a3-3119-4b1a-a642-9a5c7c21ff5a" (UID: "97b887a3-3119-4b1a-a642-9a5c7c21ff5a"). InnerVolumeSpecName "kube-api-access-lkhfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:07:25.493263 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:25.493227    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkhfz\" (UniqueName: \"kubernetes.io/projected/97b887a3-3119-4b1a-a642-9a5c7c21ff5a-kube-api-access-lkhfz\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:25.493263 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:25.493259    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b887a3-3119-4b1a-a642-9a5c7c21ff5a-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:25.881779 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:07:25.881742    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:07:26.029918 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:26.029881    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq" event={"ID":"8a586fea-87b6-4599-beed-36aba543e3c9","Type":"ContainerStarted","Data":"cf82a2b426f0540c81393b87b754f301d30e18651a20870676b7824dc59b6a21"}
Nov 01 06:07:26.030117 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:26.029983    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq"
Nov 01 06:07:26.031350 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:26.031314    5619 generic.go:356] "Generic (PLEG): container finished" podID="97b887a3-3119-4b1a-a642-9a5c7c21ff5a" containerID="febabc79ba6c36db7ac1d2c0bea7045c9f9c8ed0be52353c2d278bd9c950ac99" exitCode=0
Nov 01 06:07:26.031462 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:26.031367    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns" event={"ID":"97b887a3-3119-4b1a-a642-9a5c7c21ff5a","Type":"ContainerDied","Data":"febabc79ba6c36db7ac1d2c0bea7045c9f9c8ed0be52353c2d278bd9c950ac99"}
Nov 01 06:07:26.031462 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:26.031377    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns"
Nov 01 06:07:26.031462 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:26.031399    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns" event={"ID":"97b887a3-3119-4b1a-a642-9a5c7c21ff5a","Type":"ContainerDied","Data":"fa61a027ea09182966bb6a061e2e924fdf80af8346eefd1776731f25839790cb"}
Nov 01 06:07:26.031462 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:26.031420    5619 scope.go:122] "RemoveContainer" containerID="febabc79ba6c36db7ac1d2c0bea7045c9f9c8ed0be52353c2d278bd9c950ac99"
Nov 01 06:07:26.041584 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:26.041566    5619 scope.go:122] "RemoveContainer" containerID="febabc79ba6c36db7ac1d2c0bea7045c9f9c8ed0be52353c2d278bd9c950ac99"
Nov 01 06:07:26.041844 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:07:26.041824    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"febabc79ba6c36db7ac1d2c0bea7045c9f9c8ed0be52353c2d278bd9c950ac99\": container with ID starting with febabc79ba6c36db7ac1d2c0bea7045c9f9c8ed0be52353c2d278bd9c950ac99 not found: ID does not exist" containerID="febabc79ba6c36db7ac1d2c0bea7045c9f9c8ed0be52353c2d278bd9c950ac99"
Nov 01 06:07:26.041900 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:26.041852    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"febabc79ba6c36db7ac1d2c0bea7045c9f9c8ed0be52353c2d278bd9c950ac99"} err="failed to get container status \"febabc79ba6c36db7ac1d2c0bea7045c9f9c8ed0be52353c2d278bd9c950ac99\": rpc error: code = NotFound desc = could not find container \"febabc79ba6c36db7ac1d2c0bea7045c9f9c8ed0be52353c2d278bd9c950ac99\": container with ID starting with febabc79ba6c36db7ac1d2c0bea7045c9f9c8ed0be52353c2d278bd9c950ac99 not found: ID does not exist"
Nov 01 06:07:26.148672 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:26.148568    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq" podStartSLOduration=1.873248121 podStartE2EDuration="4.148554741s" totalImagesPullingTime="2.27530662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:07:26.147682644 +0000 UTC m=+1283.358506153" watchObservedRunningTime="2026-11-01 06:07:26.148554741 +0000 UTC m=+1283.359378249"
Nov 01 06:07:26.261602 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:26.261567    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns"]
Nov 01 06:07:26.273397 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:26.273367    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0d2807-849fcb9595-4jcns"]
Nov 01 06:07:26.627506 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:26.624302    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:27.047717 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:27.047051    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97b887a3-3119-4b1a-a642-9a5c7c21ff5a" path="/var/lib/kubelet/pods/97b887a3-3119-4b1a-a642-9a5c7c21ff5a/volumes"
Nov 01 06:07:29.378020 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:29.377984    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:33.215693 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.215658    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"]
Nov 01 06:07:33.216228 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.216215    5619 cpu_manager.go:414] "containerMap: removing container" podUID="680af3cd-644f-459b-81bf-686fdc62b198" containerName="istio-proxy"
Nov 01 06:07:33.216271 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.216230    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680af3cd-644f-459b-81bf-686fdc62b198" containerName="istio-proxy"
Nov 01 06:07:33.216271 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.216243    5619 cpu_manager.go:414] "containerMap: removing container" podUID="97b887a3-3119-4b1a-a642-9a5c7c21ff5a" containerName="payload-processing"
Nov 01 06:07:33.216271 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.216248    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b887a3-3119-4b1a-a642-9a5c7c21ff5a" containerName="payload-processing"
Nov 01 06:07:33.216392 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.216346    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="680af3cd-644f-459b-81bf-686fdc62b198" containerName="istio-proxy"
Nov 01 06:07:33.216392 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.216364    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b887a3-3119-4b1a-a642-9a5c7c21ff5a" containerName="payload-processing"
Nov 01 06:07:33.219542 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.219518    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:33.273258 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.273230    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-58d523\"/\"openshift-service-ca.crt\""
Nov 01 06:07:33.273488 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.273469    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-58d523\"/\"ipp-route-58d523-kserve-self-signed-certs\""
Nov 01 06:07:33.310712 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.310680    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:33.324527 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.324496    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-58d523\"/\"kube-root-ca.crt\""
Nov 01 06:07:33.324666 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.324539    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-58d523\"/\"default-dockercfg-84xsz\""
Nov 01 06:07:33.327178 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.327155    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"]
Nov 01 06:07:33.349248 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.349218    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-model-cache\") pod \"ipp-route-58d523-kserve-75f8cf4b45-9zfmq\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") " pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:33.349417 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.349372    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-home\") pod \"ipp-route-58d523-kserve-75f8cf4b45-9zfmq\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") " pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:33.349417 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.349399    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blkkp\" (UniqueName: \"kubernetes.io/projected/1c40af1b-82e2-4a63-a08b-b371bc511bb2-kube-api-access-blkkp\") pod \"ipp-route-58d523-kserve-75f8cf4b45-9zfmq\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") " pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:33.349544 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.349531    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-dshm\") pod \"ipp-route-58d523-kserve-75f8cf4b45-9zfmq\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") " pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:33.349602 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.349557    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-tmp-dir\") pod \"ipp-route-58d523-kserve-75f8cf4b45-9zfmq\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") " pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:33.349653 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.349618    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c40af1b-82e2-4a63-a08b-b371bc511bb2-tls-certs\") pod \"ipp-route-58d523-kserve-75f8cf4b45-9zfmq\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") " pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:33.450464 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.450426    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-home\") pod \"ipp-route-58d523-kserve-75f8cf4b45-9zfmq\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") " pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:33.450464 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.450468    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blkkp\" (UniqueName: \"kubernetes.io/projected/1c40af1b-82e2-4a63-a08b-b371bc511bb2-kube-api-access-blkkp\") pod \"ipp-route-58d523-kserve-75f8cf4b45-9zfmq\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") " pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:33.450730 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.450611    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-dshm\") pod \"ipp-route-58d523-kserve-75f8cf4b45-9zfmq\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") " pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:33.450730 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.450652    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-tmp-dir\") pod \"ipp-route-58d523-kserve-75f8cf4b45-9zfmq\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") " pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:33.450730 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.450710    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c40af1b-82e2-4a63-a08b-b371bc511bb2-tls-certs\") pod \"ipp-route-58d523-kserve-75f8cf4b45-9zfmq\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") " pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:33.450908 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.450785    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-model-cache\") pod \"ipp-route-58d523-kserve-75f8cf4b45-9zfmq\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") " pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:33.450908 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.450890    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-home\") pod \"ipp-route-58d523-kserve-75f8cf4b45-9zfmq\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") " pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:33.451164 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.451137    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-model-cache\") pod \"ipp-route-58d523-kserve-75f8cf4b45-9zfmq\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") " pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:33.451336 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.451304    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-tmp-dir\") pod \"ipp-route-58d523-kserve-75f8cf4b45-9zfmq\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") " pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:33.453601 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.453573    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-dshm\") pod \"ipp-route-58d523-kserve-75f8cf4b45-9zfmq\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") " pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:33.453881 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.453863    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c40af1b-82e2-4a63-a08b-b371bc511bb2-tls-certs\") pod \"ipp-route-58d523-kserve-75f8cf4b45-9zfmq\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") " pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:33.512848 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.512776    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blkkp\" (UniqueName: \"kubernetes.io/projected/1c40af1b-82e2-4a63-a08b-b371bc511bb2-kube-api-access-blkkp\") pod \"ipp-route-58d523-kserve-75f8cf4b45-9zfmq\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") " pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:33.531377 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.531313    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:33.911750 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:33.911725    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"]
Nov 01 06:07:33.913213 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:07:33.913186    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c40af1b_82e2_4a63_a08b_b371bc511bb2.slice/crio-6f2768cd575e0ca19838fcf124a0a5456e6c132801d077f092bf1ec8f0b34e45 WatchSource:0}: Error finding container 6f2768cd575e0ca19838fcf124a0a5456e6c132801d077f092bf1ec8f0b34e45: Status 404 returned error can't find the container with id 6f2768cd575e0ca19838fcf124a0a5456e6c132801d077f092bf1ec8f0b34e45
Nov 01 06:07:34.138988 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:34.138940    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq" event={"ID":"1c40af1b-82e2-4a63-a08b-b371bc511bb2","Type":"ContainerStarted","Data":"7b2edc0e1c26f9acf145688b07dcd6ebaa304412ddb135f5e2bedf795b7a93cc"}
Nov 01 06:07:34.138988 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:34.138991    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq" event={"ID":"1c40af1b-82e2-4a63-a08b-b371bc511bb2","Type":"ContainerStarted","Data":"6f2768cd575e0ca19838fcf124a0a5456e6c132801d077f092bf1ec8f0b34e45"}
Nov 01 06:07:34.180744 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:34.180627    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:34.532418 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:34.532307    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:37.083042 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:37.083004    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq"
Nov 01 06:07:37.202527 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:37.202472    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq" podStartSLOduration=4.202457287 podStartE2EDuration="4.202457287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:07:34.262621132 +0000 UTC m=+1291.473444651" watchObservedRunningTime="2026-11-01 06:07:37.202457287 +0000 UTC m=+1294.413280795"
Nov 01 06:07:38.964249 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:38.964194    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:43.125099 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:43.125061    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"]
Nov 01 06:07:43.125567 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:43.125265    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7" podUID="04cd4917-baa4-421d-9638-d66d4b93ca79" containerName="maas-api" containerID="cri-o://143c9f260c88167a8dd1febac339ca7daf98f23b4ec6feff13b8ea75c7ea7e94" gracePeriod=30
Nov 01 06:07:43.224409 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:43.224371    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:43.436384 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:43.436358    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"
Nov 01 06:07:43.532413 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:43.532371    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:43.533425 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:43.533405    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04cd4917-baa4-421d-9638-d66d4b93ca79-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cd4917-baa4-421d-9638-d66d4b93ca79-openshift-service-ca\") pod \"04cd4917-baa4-421d-9638-d66d4b93ca79\" (UID: \"04cd4917-baa4-421d-9638-d66d4b93ca79\") "
Nov 01 06:07:43.533557 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:43.533540    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04cd4917-baa4-421d-9638-d66d4b93ca79-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04cd4917-baa4-421d-9638-d66d4b93ca79-maas-api-tls\") pod \"04cd4917-baa4-421d-9638-d66d4b93ca79\" (UID: \"04cd4917-baa4-421d-9638-d66d4b93ca79\") "
Nov 01 06:07:43.533637 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:43.533581    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04cd4917-baa4-421d-9638-d66d4b93ca79-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04cd4917-baa4-421d-9638-d66d4b93ca79-metrics-certs\") pod \"04cd4917-baa4-421d-9638-d66d4b93ca79\" (UID: \"04cd4917-baa4-421d-9638-d66d4b93ca79\") "
Nov 01 06:07:43.533701 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:43.533653    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04cd4917-baa4-421d-9638-d66d4b93ca79-kube-api-access-mfd6m\" (UniqueName: \"kubernetes.io/projected/04cd4917-baa4-421d-9638-d66d4b93ca79-kube-api-access-mfd6m\") pod \"04cd4917-baa4-421d-9638-d66d4b93ca79\" (UID: \"04cd4917-baa4-421d-9638-d66d4b93ca79\") "
Nov 01 06:07:43.533847 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:43.533817    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04cd4917-baa4-421d-9638-d66d4b93ca79-openshift-service-ca" pod "04cd4917-baa4-421d-9638-d66d4b93ca79" (UID: "04cd4917-baa4-421d-9638-d66d4b93ca79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:07:43.534067 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:43.534047    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cd4917-baa4-421d-9638-d66d4b93ca79-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:43.535986 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:43.535962    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04cd4917-baa4-421d-9638-d66d4b93ca79-metrics-certs" pod "04cd4917-baa4-421d-9638-d66d4b93ca79" (UID: "04cd4917-baa4-421d-9638-d66d4b93ca79"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:07:43.536086 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:43.536003    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04cd4917-baa4-421d-9638-d66d4b93ca79-maas-api-tls" pod "04cd4917-baa4-421d-9638-d66d4b93ca79" (UID: "04cd4917-baa4-421d-9638-d66d4b93ca79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:07:43.536202 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:43.536185    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04cd4917-baa4-421d-9638-d66d4b93ca79-kube-api-access-mfd6m" pod "04cd4917-baa4-421d-9638-d66d4b93ca79" (UID: "04cd4917-baa4-421d-9638-d66d4b93ca79"). InnerVolumeSpecName "kube-api-access-mfd6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:07:43.545875 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:43.545855    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:43.635756 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:43.635728    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04cd4917-baa4-421d-9638-d66d4b93ca79-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:43.635756 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:43.635752    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04cd4917-baa4-421d-9638-d66d4b93ca79-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:43.635756 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:43.635765    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfd6m\" (UniqueName: \"kubernetes.io/projected/04cd4917-baa4-421d-9638-d66d4b93ca79-kube-api-access-mfd6m\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:44.216102 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:44.216067    5619 generic.go:356] "Generic (PLEG): container finished" podID="04cd4917-baa4-421d-9638-d66d4b93ca79" containerID="143c9f260c88167a8dd1febac339ca7daf98f23b4ec6feff13b8ea75c7ea7e94" exitCode=0
Nov 01 06:07:44.216583 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:44.216132    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"
Nov 01 06:07:44.216583 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:44.216162    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7" event={"ID":"04cd4917-baa4-421d-9638-d66d4b93ca79","Type":"ContainerDied","Data":"143c9f260c88167a8dd1febac339ca7daf98f23b4ec6feff13b8ea75c7ea7e94"}
Nov 01 06:07:44.216583 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:44.216221    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7" event={"ID":"04cd4917-baa4-421d-9638-d66d4b93ca79","Type":"ContainerDied","Data":"3eb1b957f0739b69a483834853fbe510b26ac8b610060ba98701ad72df666619"}
Nov 01 06:07:44.216583 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:44.216244    5619 scope.go:122] "RemoveContainer" containerID="143c9f260c88167a8dd1febac339ca7daf98f23b4ec6feff13b8ea75c7ea7e94"
Nov 01 06:07:44.233612 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:44.233590    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:07:44.235377 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:44.235359    5619 scope.go:122] "RemoveContainer" containerID="143c9f260c88167a8dd1febac339ca7daf98f23b4ec6feff13b8ea75c7ea7e94"
Nov 01 06:07:44.235657 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:07:44.235638    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"143c9f260c88167a8dd1febac339ca7daf98f23b4ec6feff13b8ea75c7ea7e94\": container with ID starting with 143c9f260c88167a8dd1febac339ca7daf98f23b4ec6feff13b8ea75c7ea7e94 not found: ID does not exist" containerID="143c9f260c88167a8dd1febac339ca7daf98f23b4ec6feff13b8ea75c7ea7e94"
Nov 01 06:07:44.235706 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:44.235665    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"143c9f260c88167a8dd1febac339ca7daf98f23b4ec6feff13b8ea75c7ea7e94"} err="failed to get container status \"143c9f260c88167a8dd1febac339ca7daf98f23b4ec6feff13b8ea75c7ea7e94\": rpc error: code = NotFound desc = could not find container \"143c9f260c88167a8dd1febac339ca7daf98f23b4ec6feff13b8ea75c7ea7e94\": container with ID starting with 143c9f260c88167a8dd1febac339ca7daf98f23b4ec6feff13b8ea75c7ea7e94 not found: ID does not exist"
Nov 01 06:07:44.507726 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:44.507612    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"]
Nov 01 06:07:44.510028 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:44.510001    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cda04f-74548c7b7f-hfrq7"]
Nov 01 06:07:44.964434 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:44.964394    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04cd4917-baa4-421d-9638-d66d4b93ca79" path="/var/lib/kubelet/pods/04cd4917-baa4-421d-9638-d66d4b93ca79/volumes"
Nov 01 06:07:44.974460 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:44.974425    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:46.485894 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:46.485853    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:49.961043 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:49.960995    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:50.155544 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:50.155507    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:51.775629 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:51.775585    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:52.319285 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:52.319253    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"]
Nov 01 06:07:52.319511 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:52.319488    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb" podUID="a1bfd841-06b0-48b4-9037-d6c77ddf0521" containerName="istio-proxy" containerID="cri-o://ac28211bf0c9792bfa38fb5dabb7175c47be643202213beff2f76f9dce24c6f2" gracePeriod=30
Nov 01 06:07:52.322115 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:52.322081    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:53.915609 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:53.915566    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:55.879768 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:07:55.879724    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:07:56.040612 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.040579    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"]
Nov 01 06:07:56.041154 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.041131    5619 cpu_manager.go:414] "containerMap: removing container" podUID="04cd4917-baa4-421d-9638-d66d4b93ca79" containerName="maas-api"
Nov 01 06:07:56.041154 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.041147    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04cd4917-baa4-421d-9638-d66d4b93ca79" containerName="maas-api"
Nov 01 06:07:56.041370 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.041247    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="04cd4917-baa4-421d-9638-d66d4b93ca79" containerName="maas-api"
Nov 01 06:07:56.044815 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.044790    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.098378 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.098313    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-3578ce-openshift-default-dockercfg-mrrt4\""
Nov 01 06:07:56.126582 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.126555    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:56.152444 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.152350    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"]
Nov 01 06:07:56.228479 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.228443    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-envoy\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.228797 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.228757    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-token\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.228924 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.228809    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-workload-certs\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.228924 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.228883    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-podinfo\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.229110 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.228932    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-data\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.229110 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.228972    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-workload-socket\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.229110 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.229045    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vwbm\" (UniqueName: \"kubernetes.io/projected/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-kube-api-access-7vwbm\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.229110 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.229083    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-credential-socket\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.229366 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.229280    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istiod-ca-cert\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.291036 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.291000    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:56.330933 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.330896    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-data\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.331118 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.330958    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-workload-socket\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.331118 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.331033    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vwbm\" (UniqueName: \"kubernetes.io/projected/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-kube-api-access-7vwbm\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.331118 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.331072    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-credential-socket\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.331275 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.331147    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istiod-ca-cert\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.331275 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.331251    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-envoy\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.331408 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.331283    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-token\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.331408 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.331310    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-data\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.331408 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.331340    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-workload-socket\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.331408 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.331317    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-workload-certs\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.331648 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.331445    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-credential-socket\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.331648 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.331518    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-podinfo\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.331773 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.331754    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-workload-certs\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.332090 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.332068    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istiod-ca-cert\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.333957 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.333938    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-envoy\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.334127 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.334108    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-podinfo\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.392907 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.392879    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-token\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.393024 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.392977    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vwbm\" (UniqueName: \"kubernetes.io/projected/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-kube-api-access-7vwbm\") pod \"e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") " pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.659048 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.659014    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:56.755816 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:56.755779    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:07:57.089996 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.089967    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"]
Nov 01 06:07:57.092336 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:07:57.092282    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3e9603_cafa_487a_9b7b_9b6b5ea7ebea.slice/crio-e838ed9b57ade69fc88e68599424fe8afadf723e66d9a3a67658e1ca6d35809f WatchSource:0}: Error finding container e838ed9b57ade69fc88e68599424fe8afadf723e66d9a3a67658e1ca6d35809f: Status 404 returned error can't find the container with id e838ed9b57ade69fc88e68599424fe8afadf723e66d9a3a67658e1ca6d35809f
Nov 01 06:07:57.095257 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.095222    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:07:57.095385 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.095286    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:07:57.095385 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.095314    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:07:57.338404 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.338361    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx" event={"ID":"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea","Type":"ContainerStarted","Data":"d2bd719db01ac1dcbd2a8a8419906442d87c59664861e21ab80063beae02eac5"}
Nov 01 06:07:57.338574 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.338435    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx" event={"ID":"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea","Type":"ContainerStarted","Data":"e838ed9b57ade69fc88e68599424fe8afadf723e66d9a3a67658e1ca6d35809f"}
Nov 01 06:07:57.377300 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.377262    5619 patch_prober.go:29] interesting pod/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 01 06:07:57.377454 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.377342    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb" podUID="a1bfd841-06b0-48b4-9037-d6c77ddf0521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 01 06:07:57.465245 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.464687    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx" podStartSLOduration=2.464668665 podStartE2EDuration="2.464668665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:07:57.461032558 +0000 UTC m=+1314.671856072" watchObservedRunningTime="2026-11-01 06:07:57.464668665 +0000 UTC m=+1314.675492176"
Nov 01 06:07:57.659929 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.659904    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:57.660266 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.660245    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:07:57.749823 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.749792    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-workload-socket\") pod \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") "
Nov 01 06:07:57.749983 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.749860    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1bfd841-06b0-48b4-9037-d6c77ddf0521-kube-api-access-nbm9k\" (UniqueName: \"kubernetes.io/projected/a1bfd841-06b0-48b4-9037-d6c77ddf0521-kube-api-access-nbm9k\") pod \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") "
Nov 01 06:07:57.750033 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.749998    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-credential-socket\") pod \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") "
Nov 01 06:07:57.750033 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.750027    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-token\" (UniqueName: \"kubernetes.io/projected/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-token\") pod \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") "
Nov 01 06:07:57.750165 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.750065    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-podinfo\") pod \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") "
Nov 01 06:07:57.750165 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.750091    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-workload-socket" pod "a1bfd841-06b0-48b4-9037-d6c77ddf0521" (UID: "a1bfd841-06b0-48b4-9037-d6c77ddf0521"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:07:57.750165 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.750107    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-data\") pod \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") "
Nov 01 06:07:57.750347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.750191    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-envoy\") pod \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") "
Nov 01 06:07:57.750347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.750280    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-workload-certs\") pod \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") "
Nov 01 06:07:57.750470 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.750387    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istiod-ca-cert\") pod \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\" (UID: \"a1bfd841-06b0-48b4-9037-d6c77ddf0521\") "
Nov 01 06:07:57.750554 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.750528    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-credential-socket" pod "a1bfd841-06b0-48b4-9037-d6c77ddf0521" (UID: "a1bfd841-06b0-48b4-9037-d6c77ddf0521"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:07:57.750977 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.750944    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:57.750977 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.750968    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:57.751256 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.751235    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-data" pod "a1bfd841-06b0-48b4-9037-d6c77ddf0521" (UID: "a1bfd841-06b0-48b4-9037-d6c77ddf0521"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:07:57.751477 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.751443    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-workload-certs" pod "a1bfd841-06b0-48b4-9037-d6c77ddf0521" (UID: "a1bfd841-06b0-48b4-9037-d6c77ddf0521"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:07:57.751759 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.751740    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istiod-ca-cert" pod "a1bfd841-06b0-48b4-9037-d6c77ddf0521" (UID: "a1bfd841-06b0-48b4-9037-d6c77ddf0521"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:07:57.752534 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.752494    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-token" pod "a1bfd841-06b0-48b4-9037-d6c77ddf0521" (UID: "a1bfd841-06b0-48b4-9037-d6c77ddf0521"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:07:57.752637 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.752579    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1bfd841-06b0-48b4-9037-d6c77ddf0521-kube-api-access-nbm9k" pod "a1bfd841-06b0-48b4-9037-d6c77ddf0521" (UID: "a1bfd841-06b0-48b4-9037-d6c77ddf0521"). InnerVolumeSpecName "kube-api-access-nbm9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:07:57.752705 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.752683    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-podinfo" pod "a1bfd841-06b0-48b4-9037-d6c77ddf0521" (UID: "a1bfd841-06b0-48b4-9037-d6c77ddf0521"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:07:57.753029 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.753008    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-envoy" pod "a1bfd841-06b0-48b4-9037-d6c77ddf0521" (UID: "a1bfd841-06b0-48b4-9037-d6c77ddf0521"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:07:57.851870 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.851832    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:57.851870 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.851864    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbm9k\" (UniqueName: \"kubernetes.io/projected/a1bfd841-06b0-48b4-9037-d6c77ddf0521-kube-api-access-nbm9k\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:57.852078 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.851882    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:57.852078 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.851896    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:57.852078 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.851908    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:57.852078 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.851921    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:57.852078 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:57.851934    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bfd841-06b0-48b4-9037-d6c77ddf0521-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:07:58.345496 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:58.345464    5619 generic.go:356] "Generic (PLEG): container finished" podID="a1bfd841-06b0-48b4-9037-d6c77ddf0521" containerID="ac28211bf0c9792bfa38fb5dabb7175c47be643202213beff2f76f9dce24c6f2" exitCode=0
Nov 01 06:07:58.345957 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:58.345514    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"
Nov 01 06:07:58.345957 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:58.345540    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb" event={"ID":"a1bfd841-06b0-48b4-9037-d6c77ddf0521","Type":"ContainerDied","Data":"ac28211bf0c9792bfa38fb5dabb7175c47be643202213beff2f76f9dce24c6f2"}
Nov 01 06:07:58.345957 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:58.345602    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb" event={"ID":"a1bfd841-06b0-48b4-9037-d6c77ddf0521","Type":"ContainerDied","Data":"30f5ea7673460f47f29508a1eb982b15914af8a58178def61f896c2c61e62cdf"}
Nov 01 06:07:58.345957 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:58.345634    5619 scope.go:122] "RemoveContainer" containerID="ac28211bf0c9792bfa38fb5dabb7175c47be643202213beff2f76f9dce24c6f2"
Nov 01 06:07:58.358787 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:58.356928    5619 scope.go:122] "RemoveContainer" containerID="71ae52201a399b1f9a8bdb6a54ea7c2467a605b14455cac50651193653f5012a"
Nov 01 06:07:58.372018 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:58.371985    5619 scope.go:122] "RemoveContainer" containerID="ac28211bf0c9792bfa38fb5dabb7175c47be643202213beff2f76f9dce24c6f2"
Nov 01 06:07:58.372303 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:07:58.372283    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac28211bf0c9792bfa38fb5dabb7175c47be643202213beff2f76f9dce24c6f2\": container with ID starting with ac28211bf0c9792bfa38fb5dabb7175c47be643202213beff2f76f9dce24c6f2 not found: ID does not exist" containerID="ac28211bf0c9792bfa38fb5dabb7175c47be643202213beff2f76f9dce24c6f2"
Nov 01 06:07:58.372400 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:58.372317    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac28211bf0c9792bfa38fb5dabb7175c47be643202213beff2f76f9dce24c6f2"} err="failed to get container status \"ac28211bf0c9792bfa38fb5dabb7175c47be643202213beff2f76f9dce24c6f2\": rpc error: code = NotFound desc = could not find container \"ac28211bf0c9792bfa38fb5dabb7175c47be643202213beff2f76f9dce24c6f2\": container with ID starting with ac28211bf0c9792bfa38fb5dabb7175c47be643202213beff2f76f9dce24c6f2 not found: ID does not exist"
Nov 01 06:07:58.372400 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:58.372356    5619 scope.go:122] "RemoveContainer" containerID="71ae52201a399b1f9a8bdb6a54ea7c2467a605b14455cac50651193653f5012a"
Nov 01 06:07:58.372645 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:07:58.372628    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ae52201a399b1f9a8bdb6a54ea7c2467a605b14455cac50651193653f5012a\": container with ID starting with 71ae52201a399b1f9a8bdb6a54ea7c2467a605b14455cac50651193653f5012a not found: ID does not exist" containerID="71ae52201a399b1f9a8bdb6a54ea7c2467a605b14455cac50651193653f5012a"
Nov 01 06:07:58.372708 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:58.372653    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ae52201a399b1f9a8bdb6a54ea7c2467a605b14455cac50651193653f5012a"} err="failed to get container status \"71ae52201a399b1f9a8bdb6a54ea7c2467a605b14455cac50651193653f5012a\": rpc error: code = NotFound desc = could not find container \"71ae52201a399b1f9a8bdb6a54ea7c2467a605b14455cac50651193653f5012a\": container with ID starting with 71ae52201a399b1f9a8bdb6a54ea7c2467a605b14455cac50651193653f5012a not found: ID does not exist"
Nov 01 06:07:58.514176 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:58.514141    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"]
Nov 01 06:07:58.566517 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:58.566474    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-cda04f-openshift-default-5f4564d67c-6t4nb"]
Nov 01 06:07:58.665551 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:58.665467    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:58.963075 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:58.962987    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1bfd841-06b0-48b4-9037-d6c77ddf0521" path="/var/lib/kubelet/pods/a1bfd841-06b0-48b4-9037-d6c77ddf0521/volumes"
Nov 01 06:07:59.353366 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:59.353340    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:07:59.354299 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:07:59.354277    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:08:00.319503 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:00.319466    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:00.569627 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:00.569546    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:00.720936 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:00.720900    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:02.140921 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:02.140881    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:03.660970 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:03.660923    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:03.837826 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:03.837733    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:04.227557 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.227526    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh"]
Nov 01 06:08:04.228053 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.228039    5619 cpu_manager.go:414] "containerMap: removing container" podUID="a1bfd841-06b0-48b4-9037-d6c77ddf0521" containerName="istio-proxy"
Nov 01 06:08:04.228098 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.228054    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1bfd841-06b0-48b4-9037-d6c77ddf0521" containerName="istio-proxy"
Nov 01 06:08:04.228098 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.228071    5619 cpu_manager.go:414] "containerMap: removing container" podUID="a1bfd841-06b0-48b4-9037-d6c77ddf0521" containerName="istio-proxy"
Nov 01 06:08:04.228098 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.228076    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1bfd841-06b0-48b4-9037-d6c77ddf0521" containerName="istio-proxy"
Nov 01 06:08:04.228190 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.228175    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1bfd841-06b0-48b4-9037-d6c77ddf0521" containerName="istio-proxy"
Nov 01 06:08:04.228190 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.228184    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1bfd841-06b0-48b4-9037-d6c77ddf0521" containerName="istio-proxy"
Nov 01 06:08:04.233148 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.233128    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh"
Nov 01 06:08:04.269220 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.269185    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e66b197-c227-414d-8fa8-8f39a228fabd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh\" (UID: \"0e66b197-c227-414d-8fa8-8f39a228fabd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh"
Nov 01 06:08:04.269362 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.269265    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcds7\" (UniqueName: \"kubernetes.io/projected/0e66b197-c227-414d-8fa8-8f39a228fabd-kube-api-access-vcds7\") pod \"payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh\" (UID: \"0e66b197-c227-414d-8fa8-8f39a228fabd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh"
Nov 01 06:08:04.337858 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.337734    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-3578ce\""
Nov 01 06:08:04.337858 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.337771    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-3578ce-dockercfg-g6cvf\""
Nov 01 06:08:04.340644 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.340614    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh"]
Nov 01 06:08:04.370624 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.370590    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e66b197-c227-414d-8fa8-8f39a228fabd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh\" (UID: \"0e66b197-c227-414d-8fa8-8f39a228fabd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh"
Nov 01 06:08:04.371285 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.371267    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e66b197-c227-414d-8fa8-8f39a228fabd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh\" (UID: \"0e66b197-c227-414d-8fa8-8f39a228fabd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh"
Nov 01 06:08:04.371409 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.371378    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcds7\" (UniqueName: \"kubernetes.io/projected/0e66b197-c227-414d-8fa8-8f39a228fabd-kube-api-access-vcds7\") pod \"payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh\" (UID: \"0e66b197-c227-414d-8fa8-8f39a228fabd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh"
Nov 01 06:08:04.404446 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.404415    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd"]
Nov 01 06:08:04.408515 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.408495    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd"
Nov 01 06:08:04.433057 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.433033    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcds7\" (UniqueName: \"kubernetes.io/projected/0e66b197-c227-414d-8fa8-8f39a228fabd-kube-api-access-vcds7\") pod \"payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh\" (UID: \"0e66b197-c227-414d-8fa8-8f39a228fabd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh"
Nov 01 06:08:04.465262 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.465232    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd"]
Nov 01 06:08:04.477358 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.472619    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7f6j\" (UniqueName: \"kubernetes.io/projected/bed3762c-2388-4654-9555-6c4cc3b817e3-kube-api-access-t7f6j\") pod \"payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd\" (UID: \"bed3762c-2388-4654-9555-6c4cc3b817e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd"
Nov 01 06:08:04.477358 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.472776    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bed3762c-2388-4654-9555-6c4cc3b817e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd\" (UID: \"bed3762c-2388-4654-9555-6c4cc3b817e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd"
Nov 01 06:08:04.547689 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.547609    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh"
Nov 01 06:08:04.573640 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.573593    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bed3762c-2388-4654-9555-6c4cc3b817e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd\" (UID: \"bed3762c-2388-4654-9555-6c4cc3b817e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd"
Nov 01 06:08:04.573874 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.573852    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7f6j\" (UniqueName: \"kubernetes.io/projected/bed3762c-2388-4654-9555-6c4cc3b817e3-kube-api-access-t7f6j\") pod \"payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd\" (UID: \"bed3762c-2388-4654-9555-6c4cc3b817e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd"
Nov 01 06:08:04.574603 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.574579    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bed3762c-2388-4654-9555-6c4cc3b817e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd\" (UID: \"bed3762c-2388-4654-9555-6c4cc3b817e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd"
Nov 01 06:08:04.636084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.636050    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7f6j\" (UniqueName: \"kubernetes.io/projected/bed3762c-2388-4654-9555-6c4cc3b817e3-kube-api-access-t7f6j\") pod \"payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd\" (UID: \"bed3762c-2388-4654-9555-6c4cc3b817e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd"
Nov 01 06:08:04.721839 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.721803    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd"
Nov 01 06:08:04.931189 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:04.931150    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh"]
Nov 01 06:08:04.932627 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:08:04.932590    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e66b197_c227_414d_8fa8_8f39a228fabd.slice/crio-15e516c399d59950145bad5b70391ff50efbf6cef96b8cbcd48caed831822ced WatchSource:0}: Error finding container 15e516c399d59950145bad5b70391ff50efbf6cef96b8cbcd48caed831822ced: Status 404 returned error can't find the container with id 15e516c399d59950145bad5b70391ff50efbf6cef96b8cbcd48caed831822ced
Nov 01 06:08:05.109498 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:05.109468    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd"]
Nov 01 06:08:05.110665 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:08:05.110638    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed3762c_2388_4654_9555_6c4cc3b817e3.slice/crio-b71341c1ef0dbd49217b9b67cae3493a3c57b66d318a3a5203a279dcf922ac06 WatchSource:0}: Error finding container b71341c1ef0dbd49217b9b67cae3493a3c57b66d318a3a5203a279dcf922ac06: Status 404 returned error can't find the container with id b71341c1ef0dbd49217b9b67cae3493a3c57b66d318a3a5203a279dcf922ac06
Nov 01 06:08:05.439823 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:05.439784    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd" event={"ID":"bed3762c-2388-4654-9555-6c4cc3b817e3","Type":"ContainerStarted","Data":"e3c90461bc8a5f65c4c7073408080b809eae2a868ac01de2705f7cb32fe437e5"}
Nov 01 06:08:05.440007 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:05.439989    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd"
Nov 01 06:08:05.440098 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:05.440028    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd" event={"ID":"bed3762c-2388-4654-9555-6c4cc3b817e3","Type":"ContainerStarted","Data":"b71341c1ef0dbd49217b9b67cae3493a3c57b66d318a3a5203a279dcf922ac06"}
Nov 01 06:08:05.441645 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:05.441621    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh" event={"ID":"0e66b197-c227-414d-8fa8-8f39a228fabd","Type":"ContainerStarted","Data":"6010e2b0db05c73f2ab95f622bde176761efdae2a1701d36fde9256df5ba4cf2"}
Nov 01 06:08:05.441764 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:05.441667    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh" event={"ID":"0e66b197-c227-414d-8fa8-8f39a228fabd","Type":"ContainerStarted","Data":"15e516c399d59950145bad5b70391ff50efbf6cef96b8cbcd48caed831822ced"}
Nov 01 06:08:05.441764 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:05.441708    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh"
Nov 01 06:08:05.559909 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:05.559849    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd" podStartSLOduration=1.559831226 podStartE2EDuration="1.559831226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:08:05.556904438 +0000 UTC m=+1322.767727947" watchObservedRunningTime="2026-11-01 06:08:05.559831226 +0000 UTC m=+1322.770654736"
Nov 01 06:08:05.695442 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:05.693781    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh" podStartSLOduration=1.693761328 podStartE2EDuration="1.693761328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:08:05.692206785 +0000 UTC m=+1322.903030295" watchObservedRunningTime="2026-11-01 06:08:05.693761328 +0000 UTC m=+1322.904584839"
Nov 01 06:08:08.773951 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:08.773912    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:13.645758 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:13.645719    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:13.818362 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:13.814376    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:14.021957 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:14.021863    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:16.450684 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:16.450655    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd"
Nov 01 06:08:16.451114 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:16.450779    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh"
Nov 01 06:08:18.491978 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:18.491938    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:18.871353 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:18.870918    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:23.277051 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:23.277016    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:25.880022 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:08:25.879982    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:08:29.852463 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:29.852419    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:32.753848 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:32.753804    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:32.977972 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:32.977933    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:33.609424 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:33.609388    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:39.035454 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:39.035419    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"]
Nov 01 06:08:39.035858 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:39.035642    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq" podUID="5d487e22-0f51-434a-98db-93e7a80b44ad" containerName="istio-proxy" containerID="cri-o://93291367d64fd20e7c4a1d82a032ac409a1be9824edb6c31329fca3206a770f6" gracePeriod=30
Nov 01 06:08:39.578176 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:39.578138    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:40.965877 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:40.965792    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:41.504419 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.504390    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"]
Nov 01 06:08:41.509671 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.509645    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.588428 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.585783    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-8f45a2-openshift-default-dockercfg-p2j5s\""
Nov 01 06:08:41.588428 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.587259    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-workload-socket\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.588428 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.587350    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-workload-certs\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.588428 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.587391    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4482833c-590d-44a1-a485-651970b9abf6-istiod-ca-cert\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.588428 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.587407    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzxxd\" (UniqueName: \"kubernetes.io/projected/4482833c-590d-44a1-a485-651970b9abf6-kube-api-access-vzxxd\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.588428 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.587474    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4482833c-590d-44a1-a485-651970b9abf6-istio-podinfo\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.588428 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.587526    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4482833c-590d-44a1-a485-651970b9abf6-istio-token\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.588428 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.587622    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-credential-socket\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.588428 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.587684    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-istio-data\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.588428 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.587755    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-istio-envoy\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.634340 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.634299    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"]
Nov 01 06:08:41.635545 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.635511    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:41.698633 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.698041    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-istio-envoy\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.698633 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.698157    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-workload-socket\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.698633 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.698206    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-workload-certs\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.698633 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.698257    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4482833c-590d-44a1-a485-651970b9abf6-istiod-ca-cert\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.698633 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.698284    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzxxd\" (UniqueName: \"kubernetes.io/projected/4482833c-590d-44a1-a485-651970b9abf6-kube-api-access-vzxxd\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.698633 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.698399    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4482833c-590d-44a1-a485-651970b9abf6-istio-podinfo\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.698633 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.698459    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4482833c-590d-44a1-a485-651970b9abf6-istio-token\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.698633 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.698516    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-credential-socket\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.699216 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.698631    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-istio-data\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.699216 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.699149    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-istio-data\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.705609 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.705582    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-istio-envoy\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.706095 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.706049    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-workload-socket\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.707501 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.707477    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-workload-certs\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.708246 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.707962    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-credential-socket\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.708519 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.708075    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4482833c-590d-44a1-a485-651970b9abf6-istiod-ca-cert\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.722466 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.722425    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4482833c-590d-44a1-a485-651970b9abf6-istio-podinfo\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.769534 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.769467    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4482833c-590d-44a1-a485-651970b9abf6-istio-token\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.769660 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.769541    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzxxd\" (UniqueName: \"kubernetes.io/projected/4482833c-590d-44a1-a485-651970b9abf6-kube-api-access-vzxxd\") pod \"e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") " pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:41.824541 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:41.824514    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:42.036443 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:42.036353    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:42.259009 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:42.258978    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"]
Nov 01 06:08:42.260785 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:08:42.260748    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4482833c_590d_44a1_a485_651970b9abf6.slice/crio-46923d1a70ce5980e89b633d70d0ded87d73c9382d11b1970a7767fd9f885c20 WatchSource:0}: Error finding container 46923d1a70ce5980e89b633d70d0ded87d73c9382d11b1970a7767fd9f885c20: Status 404 returned error can't find the container with id 46923d1a70ce5980e89b633d70d0ded87d73c9382d11b1970a7767fd9f885c20
Nov 01 06:08:42.263810 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:42.263779    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:08:42.263887 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:42.263841    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:08:42.263887 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:42.263872    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:08:42.716043 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:42.716008    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl" event={"ID":"4482833c-590d-44a1-a485-651970b9abf6","Type":"ContainerStarted","Data":"4c6426a84a6077a5ae7e36ce85a8f77889a878a31b88eb5c96419fa2a96db924"}
Nov 01 06:08:42.716207 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:42.716090    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl" event={"ID":"4482833c-590d-44a1-a485-651970b9abf6","Type":"ContainerStarted","Data":"46923d1a70ce5980e89b633d70d0ded87d73c9382d11b1970a7767fd9f885c20"}
Nov 01 06:08:42.824767 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:42.824736    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:42.840103 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:42.840049    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl" podStartSLOduration=1.8400345310000001 podStartE2EDuration="1.840034531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:08:42.837994858 +0000 UTC m=+1360.048818370" watchObservedRunningTime="2026-11-01 06:08:42.840034531 +0000 UTC m=+1360.050858042"
Nov 01 06:08:43.830895 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:43.830862    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:44.062772 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.062744    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"]
Nov 01 06:08:44.066766 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.066745    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"
Nov 01 06:08:44.120453 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.120415    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-3578ce-serving-cert\""
Nov 01 06:08:44.127330 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.127300    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"]
Nov 01 06:08:44.230768 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.230731    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19041e40-ee3c-4e90-908e-e3072a30e985-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h\" (UID: \"19041e40-ee3c-4e90-908e-e3072a30e985\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"
Nov 01 06:08:44.230938 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.230917    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj7x2\" (UniqueName: \"kubernetes.io/projected/19041e40-ee3c-4e90-908e-e3072a30e985-kube-api-access-cj7x2\") pod \"maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h\" (UID: \"19041e40-ee3c-4e90-908e-e3072a30e985\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"
Nov 01 06:08:44.231049 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.231025    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19041e40-ee3c-4e90-908e-e3072a30e985-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h\" (UID: \"19041e40-ee3c-4e90-908e-e3072a30e985\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"
Nov 01 06:08:44.231204 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.231173    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19041e40-ee3c-4e90-908e-e3072a30e985-metrics-certs\") pod \"maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h\" (UID: \"19041e40-ee3c-4e90-908e-e3072a30e985\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"
Nov 01 06:08:44.254984 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.254949    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:44.332754 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.332429    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:08:44.333000 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.332943    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj7x2\" (UniqueName: \"kubernetes.io/projected/19041e40-ee3c-4e90-908e-e3072a30e985-kube-api-access-cj7x2\") pod \"maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h\" (UID: \"19041e40-ee3c-4e90-908e-e3072a30e985\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"
Nov 01 06:08:44.333079 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.333061    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19041e40-ee3c-4e90-908e-e3072a30e985-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h\" (UID: \"19041e40-ee3c-4e90-908e-e3072a30e985\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"
Nov 01 06:08:44.333163 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.333129    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19041e40-ee3c-4e90-908e-e3072a30e985-metrics-certs\") pod \"maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h\" (UID: \"19041e40-ee3c-4e90-908e-e3072a30e985\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"
Nov 01 06:08:44.333235 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.333194    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19041e40-ee3c-4e90-908e-e3072a30e985-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h\" (UID: \"19041e40-ee3c-4e90-908e-e3072a30e985\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"
Nov 01 06:08:44.333406 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:08:44.333354    5619 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-serving-cert: secret "maas-api-e2e-praxis-a-3578ce-serving-cert" not found
Nov 01 06:08:44.333491 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:08:44.333442    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19041e40-ee3c-4e90-908e-e3072a30e985-maas-api-tls podName:19041e40-ee3c-4e90-908e-e3072a30e985 nodeName:}" failed. No retries permitted until 2026-11-01 06:08:44.833420649 +0000 UTC m=+1362.044244150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/19041e40-ee3c-4e90-908e-e3072a30e985-maas-api-tls") pod "maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h" (UID: "19041e40-ee3c-4e90-908e-e3072a30e985") : secret "maas-api-e2e-praxis-a-3578ce-serving-cert" not found
Nov 01 06:08:44.334037 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.334008    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19041e40-ee3c-4e90-908e-e3072a30e985-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h\" (UID: \"19041e40-ee3c-4e90-908e-e3072a30e985\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"
Nov 01 06:08:44.343383 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.341311    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19041e40-ee3c-4e90-908e-e3072a30e985-metrics-certs\") pod \"maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h\" (UID: \"19041e40-ee3c-4e90-908e-e3072a30e985\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"
Nov 01 06:08:44.397403 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.397298    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj7x2\" (UniqueName: \"kubernetes.io/projected/19041e40-ee3c-4e90-908e-e3072a30e985-kube-api-access-cj7x2\") pod \"maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h\" (UID: \"19041e40-ee3c-4e90-908e-e3072a30e985\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"
Nov 01 06:08:44.415957 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.415927    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:44.434543 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.434518    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-envoy\") pod \"5d487e22-0f51-434a-98db-93e7a80b44ad\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") "
Nov 01 06:08:44.434701 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.434623    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-data\") pod \"5d487e22-0f51-434a-98db-93e7a80b44ad\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") "
Nov 01 06:08:44.434770 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.434752    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d487e22-0f51-434a-98db-93e7a80b44ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d487e22-0f51-434a-98db-93e7a80b44ad-istiod-ca-cert\") pod \"5d487e22-0f51-434a-98db-93e7a80b44ad\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") "
Nov 01 06:08:44.434842 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.434824    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-credential-socket\") pod \"5d487e22-0f51-434a-98db-93e7a80b44ad\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") "
Nov 01 06:08:44.434895 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.434867    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-workload-certs\") pod \"5d487e22-0f51-434a-98db-93e7a80b44ad\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") "
Nov 01 06:08:44.434948 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.434899    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d487e22-0f51-434a-98db-93e7a80b44ad-kube-api-access-zqcgp\" (UniqueName: \"kubernetes.io/projected/5d487e22-0f51-434a-98db-93e7a80b44ad-kube-api-access-zqcgp\") pod \"5d487e22-0f51-434a-98db-93e7a80b44ad\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") "
Nov 01 06:08:44.434948 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.434932    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-podinfo\") pod \"5d487e22-0f51-434a-98db-93e7a80b44ad\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") "
Nov 01 06:08:44.435051 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.434943    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-data" pod "5d487e22-0f51-434a-98db-93e7a80b44ad" (UID: "5d487e22-0f51-434a-98db-93e7a80b44ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:44.435431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.435121    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-workload-socket\") pod \"5d487e22-0f51-434a-98db-93e7a80b44ad\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") "
Nov 01 06:08:44.435431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.435153    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-credential-socket" pod "5d487e22-0f51-434a-98db-93e7a80b44ad" (UID: "5d487e22-0f51-434a-98db-93e7a80b44ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:44.435431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.435160    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d487e22-0f51-434a-98db-93e7a80b44ad-istiod-ca-cert" pod "5d487e22-0f51-434a-98db-93e7a80b44ad" (UID: "5d487e22-0f51-434a-98db-93e7a80b44ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:08:44.435431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.435185    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-token\" (UniqueName: \"kubernetes.io/projected/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-token\") pod \"5d487e22-0f51-434a-98db-93e7a80b44ad\" (UID: \"5d487e22-0f51-434a-98db-93e7a80b44ad\") "
Nov 01 06:08:44.435431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.435242    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-workload-certs" pod "5d487e22-0f51-434a-98db-93e7a80b44ad" (UID: "5d487e22-0f51-434a-98db-93e7a80b44ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:44.435431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.435399    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-workload-socket" pod "5d487e22-0f51-434a-98db-93e7a80b44ad" (UID: "5d487e22-0f51-434a-98db-93e7a80b44ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:44.435814 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.435777    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.435814 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.435809    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d487e22-0f51-434a-98db-93e7a80b44ad-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.436008 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.435824    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.436008 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.435839    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.436008 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.435851    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.437034 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.437016    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-envoy" pod "5d487e22-0f51-434a-98db-93e7a80b44ad" (UID: "5d487e22-0f51-434a-98db-93e7a80b44ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:44.437207 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.437194    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-podinfo" pod "5d487e22-0f51-434a-98db-93e7a80b44ad" (UID: "5d487e22-0f51-434a-98db-93e7a80b44ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:08:44.437448 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.437427    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d487e22-0f51-434a-98db-93e7a80b44ad-kube-api-access-zqcgp" pod "5d487e22-0f51-434a-98db-93e7a80b44ad" (UID: "5d487e22-0f51-434a-98db-93e7a80b44ad"). InnerVolumeSpecName "kube-api-access-zqcgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:08:44.437524 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.437477    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-token" pod "5d487e22-0f51-434a-98db-93e7a80b44ad" (UID: "5d487e22-0f51-434a-98db-93e7a80b44ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:08:44.537308 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.537273    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.537308 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.537309    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.537308 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.537342    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqcgp\" (UniqueName: \"kubernetes.io/projected/5d487e22-0f51-434a-98db-93e7a80b44ad-kube-api-access-zqcgp\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.537576 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.537357    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d487e22-0f51-434a-98db-93e7a80b44ad-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:44.586390 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.572489    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:44.731100 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.731010    5619 generic.go:356] "Generic (PLEG): container finished" podID="5d487e22-0f51-434a-98db-93e7a80b44ad" containerID="93291367d64fd20e7c4a1d82a032ac409a1be9824edb6c31329fca3206a770f6" exitCode=0
Nov 01 06:08:44.731100 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.731056    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"
Nov 01 06:08:44.731100 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.731090    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq" event={"ID":"5d487e22-0f51-434a-98db-93e7a80b44ad","Type":"ContainerDied","Data":"93291367d64fd20e7c4a1d82a032ac409a1be9824edb6c31329fca3206a770f6"}
Nov 01 06:08:44.731424 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.731138    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq" event={"ID":"5d487e22-0f51-434a-98db-93e7a80b44ad","Type":"ContainerDied","Data":"35a68f4013ad42df40a269f7e7954e7cbfd17d6d7cdadfb06a99ae8dff86f68d"}
Nov 01 06:08:44.731424 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.731165    5619 scope.go:122] "RemoveContainer" containerID="93291367d64fd20e7c4a1d82a032ac409a1be9824edb6c31329fca3206a770f6"
Nov 01 06:08:44.731543 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.731510    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:44.734533 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.734508    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:08:44.745510 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.745421    5619 scope.go:122] "RemoveContainer" containerID="93291367d64fd20e7c4a1d82a032ac409a1be9824edb6c31329fca3206a770f6"
Nov 01 06:08:44.745748 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:08:44.745728    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93291367d64fd20e7c4a1d82a032ac409a1be9824edb6c31329fca3206a770f6\": container with ID starting with 93291367d64fd20e7c4a1d82a032ac409a1be9824edb6c31329fca3206a770f6 not found: ID does not exist" containerID="93291367d64fd20e7c4a1d82a032ac409a1be9824edb6c31329fca3206a770f6"
Nov 01 06:08:44.745814 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.745769    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93291367d64fd20e7c4a1d82a032ac409a1be9824edb6c31329fca3206a770f6"} err="failed to get container status \"93291367d64fd20e7c4a1d82a032ac409a1be9824edb6c31329fca3206a770f6\": rpc error: code = NotFound desc = could not find container \"93291367d64fd20e7c4a1d82a032ac409a1be9824edb6c31329fca3206a770f6\": container with ID starting with 93291367d64fd20e7c4a1d82a032ac409a1be9824edb6c31329fca3206a770f6 not found: ID does not exist"
Nov 01 06:08:44.841379 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.841313    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19041e40-ee3c-4e90-908e-e3072a30e985-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h\" (UID: \"19041e40-ee3c-4e90-908e-e3072a30e985\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"
Nov 01 06:08:44.843637 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.843613    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19041e40-ee3c-4e90-908e-e3072a30e985-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h\" (UID: \"19041e40-ee3c-4e90-908e-e3072a30e985\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"
Nov 01 06:08:44.901866 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.901834    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"]
Nov 01 06:08:44.952623 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.952591    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-snhvq"]
Nov 01 06:08:44.962248 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:44.962223    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d487e22-0f51-434a-98db-93e7a80b44ad" path="/var/lib/kubelet/pods/5d487e22-0f51-434a-98db-93e7a80b44ad/volumes"
Nov 01 06:08:45.045422 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:45.045345    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"
Nov 01 06:08:45.431766 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:45.431738    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"]
Nov 01 06:08:45.433143 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:08:45.433116    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19041e40_ee3c_4e90_908e_e3072a30e985.slice/crio-57eb3108f6bf597b568909b8592b7e118d6eb23e790454436b5bad02bb4c9abe WatchSource:0}: Error finding container 57eb3108f6bf597b568909b8592b7e118d6eb23e790454436b5bad02bb4c9abe: Status 404 returned error can't find the container with id 57eb3108f6bf597b568909b8592b7e118d6eb23e790454436b5bad02bb4c9abe
Nov 01 06:08:45.738106 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:45.738019    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h" event={"ID":"19041e40-ee3c-4e90-908e-e3072a30e985","Type":"ContainerStarted","Data":"57eb3108f6bf597b568909b8592b7e118d6eb23e790454436b5bad02bb4c9abe"}
Nov 01 06:08:46.747654 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:46.747617    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h" event={"ID":"19041e40-ee3c-4e90-908e-e3072a30e985","Type":"ContainerStarted","Data":"e207e20a59f25d4388dad84246b6842470951b78881d5d9a3f6fa3cd56dbed4c"}
Nov 01 06:08:46.748148 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:46.747931    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"
Nov 01 06:08:46.868873 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:46.868822    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h" podStartSLOduration=2.026791913 podStartE2EDuration="2.868808697s" totalImagesPullingTime="842.016784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:08:46.867209853 +0000 UTC m=+1364.078033387" watchObservedRunningTime="2026-11-01 06:08:46.868808697 +0000 UTC m=+1364.079632205"
Nov 01 06:08:49.368241 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:49.368201    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:52.759387 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:52.759347    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"
Nov 01 06:08:54.023998 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:54.023965    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"]
Nov 01 06:08:54.024467 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:54.024154    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj" podUID="e4c10775-05ae-437a-bb34-422cefafc913" containerName="istio-proxy" containerID="cri-o://d8a1230f940aec93b2736675e9e8d2288597478a29430e824c91e41b24f23e15" gracePeriod=30
Nov 01 06:08:54.197541 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:54.197503    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:54.395857 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:54.395821    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:55.907361 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:08:55.907291    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:08:56.616239 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:56.616196    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:57.442999 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:57.442959    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:57.661348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:57.654614    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:08:59.402824 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.402799    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:08:59.415309 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.415258    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c10775-05ae-437a-bb34-422cefafc913-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c10775-05ae-437a-bb34-422cefafc913-istiod-ca-cert\") pod \"e4c10775-05ae-437a-bb34-422cefafc913\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") "
Nov 01 06:08:59.415309 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.415307    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c10775-05ae-437a-bb34-422cefafc913-kube-api-access-hwvk9\" (UniqueName: \"kubernetes.io/projected/e4c10775-05ae-437a-bb34-422cefafc913-kube-api-access-hwvk9\") pod \"e4c10775-05ae-437a-bb34-422cefafc913\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") "
Nov 01 06:08:59.415592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.415372    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-istio-envoy\") pod \"e4c10775-05ae-437a-bb34-422cefafc913\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") "
Nov 01 06:08:59.415592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.415411    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-workload-certs\") pod \"e4c10775-05ae-437a-bb34-422cefafc913\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") "
Nov 01 06:08:59.415592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.415472    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c10775-05ae-437a-bb34-422cefafc913-istio-token\" (UniqueName: \"kubernetes.io/projected/e4c10775-05ae-437a-bb34-422cefafc913-istio-token\") pod \"e4c10775-05ae-437a-bb34-422cefafc913\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") "
Nov 01 06:08:59.415592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.415505    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-istio-data\") pod \"e4c10775-05ae-437a-bb34-422cefafc913\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") "
Nov 01 06:08:59.415592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.415546    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-credential-socket\") pod \"e4c10775-05ae-437a-bb34-422cefafc913\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") "
Nov 01 06:08:59.415592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.415583    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-workload-socket\") pod \"e4c10775-05ae-437a-bb34-422cefafc913\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") "
Nov 01 06:08:59.415887 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.415704    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4c10775-05ae-437a-bb34-422cefafc913-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c10775-05ae-437a-bb34-422cefafc913-istio-podinfo\") pod \"e4c10775-05ae-437a-bb34-422cefafc913\" (UID: \"e4c10775-05ae-437a-bb34-422cefafc913\") "
Nov 01 06:08:59.416672 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.416618    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-workload-certs" pod "e4c10775-05ae-437a-bb34-422cefafc913" (UID: "e4c10775-05ae-437a-bb34-422cefafc913"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:59.416792 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.416703    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-credential-socket" pod "e4c10775-05ae-437a-bb34-422cefafc913" (UID: "e4c10775-05ae-437a-bb34-422cefafc913"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:59.416792 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.416718    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-istio-data" pod "e4c10775-05ae-437a-bb34-422cefafc913" (UID: "e4c10775-05ae-437a-bb34-422cefafc913"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:59.416916 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.416815    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-workload-socket" pod "e4c10775-05ae-437a-bb34-422cefafc913" (UID: "e4c10775-05ae-437a-bb34-422cefafc913"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:59.417064 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.417044    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c10775-05ae-437a-bb34-422cefafc913-istiod-ca-cert" pod "e4c10775-05ae-437a-bb34-422cefafc913" (UID: "e4c10775-05ae-437a-bb34-422cefafc913"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:08:59.418629 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.418606    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c10775-05ae-437a-bb34-422cefafc913-istio-token" pod "e4c10775-05ae-437a-bb34-422cefafc913" (UID: "e4c10775-05ae-437a-bb34-422cefafc913"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:08:59.418850 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.418824    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c10775-05ae-437a-bb34-422cefafc913-kube-api-access-hwvk9" pod "e4c10775-05ae-437a-bb34-422cefafc913" (UID: "e4c10775-05ae-437a-bb34-422cefafc913"). InnerVolumeSpecName "kube-api-access-hwvk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:08:59.418915 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.418894    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4c10775-05ae-437a-bb34-422cefafc913-istio-podinfo" pod "e4c10775-05ae-437a-bb34-422cefafc913" (UID: "e4c10775-05ae-437a-bb34-422cefafc913"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:08:59.418953 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.418892    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-istio-envoy" pod "e4c10775-05ae-437a-bb34-422cefafc913" (UID: "e4c10775-05ae-437a-bb34-422cefafc913"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:08:59.517191 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.517153    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c10775-05ae-437a-bb34-422cefafc913-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:59.517191 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.517187    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwvk9\" (UniqueName: \"kubernetes.io/projected/e4c10775-05ae-437a-bb34-422cefafc913-kube-api-access-hwvk9\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:59.517431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.517201    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:59.517431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.517214    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:59.517431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.517226    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c10775-05ae-437a-bb34-422cefafc913-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:59.517431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.517238    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:59.517431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.517252    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:59.517431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.517265    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c10775-05ae-437a-bb34-422cefafc913-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:59.517431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.517277    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c10775-05ae-437a-bb34-422cefafc913-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:08:59.884388 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.884352    5619 generic.go:356] "Generic (PLEG): container finished" podID="e4c10775-05ae-437a-bb34-422cefafc913" containerID="d8a1230f940aec93b2736675e9e8d2288597478a29430e824c91e41b24f23e15" exitCode=0
Nov 01 06:08:59.884567 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.884456    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"
Nov 01 06:08:59.884567 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.884454    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj" event={"ID":"e4c10775-05ae-437a-bb34-422cefafc913","Type":"ContainerDied","Data":"d8a1230f940aec93b2736675e9e8d2288597478a29430e824c91e41b24f23e15"}
Nov 01 06:08:59.884567 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.884528    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj" event={"ID":"e4c10775-05ae-437a-bb34-422cefafc913","Type":"ContainerDied","Data":"ad5665ceaee04466599af5d45326f9a3dc2c97ee5538edc7d0f03ea98359dc47"}
Nov 01 06:08:59.884567 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.884561    5619 scope.go:122] "RemoveContainer" containerID="d8a1230f940aec93b2736675e9e8d2288597478a29430e824c91e41b24f23e15"
Nov 01 06:08:59.896597 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.896580    5619 scope.go:122] "RemoveContainer" containerID="d8a1230f940aec93b2736675e9e8d2288597478a29430e824c91e41b24f23e15"
Nov 01 06:08:59.896879 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:08:59.896860    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a1230f940aec93b2736675e9e8d2288597478a29430e824c91e41b24f23e15\": container with ID starting with d8a1230f940aec93b2736675e9e8d2288597478a29430e824c91e41b24f23e15 not found: ID does not exist" containerID="d8a1230f940aec93b2736675e9e8d2288597478a29430e824c91e41b24f23e15"
Nov 01 06:08:59.896953 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:08:59.896885    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a1230f940aec93b2736675e9e8d2288597478a29430e824c91e41b24f23e15"} err="failed to get container status \"d8a1230f940aec93b2736675e9e8d2288597478a29430e824c91e41b24f23e15\": rpc error: code = NotFound desc = could not find container \"d8a1230f940aec93b2736675e9e8d2288597478a29430e824c91e41b24f23e15\": container with ID starting with d8a1230f940aec93b2736675e9e8d2288597478a29430e824c91e41b24f23e15 not found: ID does not exist"
Nov 01 06:09:00.059079 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:00.059041    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"]
Nov 01 06:09:00.105507 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:00.105471    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-lqnxj"]
Nov 01 06:09:00.975006 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:00.974965    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c10775-05ae-437a-bb34-422cefafc913" path="/var/lib/kubelet/pods/e4c10775-05ae-437a-bb34-422cefafc913/volumes"
Nov 01 06:09:01.669038 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:01.668992    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:04.916867 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:04.916834    5619 generic.go:356] "Generic (PLEG): container finished" podID="7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" containerID="d2bd719db01ac1dcbd2a8a8419906442d87c59664861e21ab80063beae02eac5" exitCode=0
Nov 01 06:09:04.917445 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:04.916911    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx" event={"ID":"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea","Type":"ContainerDied","Data":"d2bd719db01ac1dcbd2a8a8419906442d87c59664861e21ab80063beae02eac5"}
Nov 01 06:09:04.917445 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:04.917440    5619 scope.go:122] "RemoveContainer" containerID="d2bd719db01ac1dcbd2a8a8419906442d87c59664861e21ab80063beae02eac5"
Nov 01 06:09:04.918914 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:04.918882    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:09:04.919007 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:04.918960    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:09:04.919007 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:04.918995    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:09:05.925782 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:05.925746    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx" event={"ID":"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea","Type":"ContainerStarted","Data":"3a0eaae25e8f8e239622ae9c98075bf543d3f9089a3e1e9abdd8648e5fa182cf"}
Nov 01 06:09:06.472342 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:06.466237    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:06.670346 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:06.666692    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:09:06.670346 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:06.666824    5619 patch_prober.go:29] interesting pod/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 01 06:09:06.670346 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:06.666882    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx" podUID="7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 01 06:09:07.665735 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:07.665694    5619 patch_prober.go:29] interesting pod/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 01 06:09:07.666409 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:07.666373    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx" podUID="7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 01 06:09:08.660718 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:08.660677    5619 patch_prober.go:29] interesting pod/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 01 06:09:08.661431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:08.661398    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx" podUID="7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 01 06:09:09.664861 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:09.664820    5619 patch_prober.go:29] interesting pod/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 01 06:09:09.665512 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:09.665481    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx" podUID="7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 01 06:09:10.664211 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:10.664173    5619 patch_prober.go:29] interesting pod/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 01 06:09:10.664527 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:10.664496    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx" podUID="7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 01 06:09:11.269986 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:11.269945    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:11.665212 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:11.665178    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:09:11.665689 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:11.665660    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:09:11.666533 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:11.666515    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:09:12.379036 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:12.378993    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:13.816291 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:13.816255    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:16.066218 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:16.066175    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:18.243897 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:18.243862    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:19.217101 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:19.217053    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:19.431006 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:19.430961    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:20.959959 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:20.959901    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:22.010027 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:22.010002    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-8444n_1f121a97-b0e6-422d-ac1b-f3a235b333e9/payload-processing/0.log"
Nov 01 06:09:22.242616 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:22.242589    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-8444n_1f121a97-b0e6-422d-ac1b-f3a235b333e9/payload-processing/0.log"
Nov 01 06:09:23.651921 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:23.651870    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:24.043424 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.043343    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"]
Nov 01 06:09:24.043973 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.043957    5619 cpu_manager.go:414] "containerMap: removing container" podUID="5d487e22-0f51-434a-98db-93e7a80b44ad" containerName="istio-proxy"
Nov 01 06:09:24.044014 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.043975    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d487e22-0f51-434a-98db-93e7a80b44ad" containerName="istio-proxy"
Nov 01 06:09:24.044048 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.044017    5619 cpu_manager.go:414] "containerMap: removing container" podUID="e4c10775-05ae-437a-bb34-422cefafc913" containerName="istio-proxy"
Nov 01 06:09:24.044048 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.044026    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c10775-05ae-437a-bb34-422cefafc913" containerName="istio-proxy"
Nov 01 06:09:24.044125 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.044116    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c10775-05ae-437a-bb34-422cefafc913" containerName="istio-proxy"
Nov 01 06:09:24.044164 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.044129    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d487e22-0f51-434a-98db-93e7a80b44ad" containerName="istio-proxy"
Nov 01 06:09:24.047459 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.047443    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.150428 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.150394    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.150616 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.150498    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.150616 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.150542    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.150616 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.150561    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.150796 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.150626    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.150796 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.150685    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wchpg\" (UniqueName: \"kubernetes.io/projected/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-kube-api-access-wchpg\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.150796 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.150730    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.150796 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.150758    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.151005 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.150799    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.155753 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.155726    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"]
Nov 01 06:09:24.252076 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.252035    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.252247 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.252092    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.252247 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.252137    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.252247 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.252198    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.252435 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.252254    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.252435 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.252304    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.252435 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.252350    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.252592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.252452    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.252592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.252506    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.252592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.252533    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.252592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.252545    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wchpg\" (UniqueName: \"kubernetes.io/projected/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-kube-api-access-wchpg\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.252801 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.252699    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.252801 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.252788    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.253203 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.253178    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.254591 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.254567    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.254841 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.254821    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.314230 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.314147    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.314396 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.314298    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wchpg\" (UniqueName: \"kubernetes.io/projected/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-kube-api-access-wchpg\") pod \"maas-default-gateway-openshift-default-855f9574cf-pskp9\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.362445 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.362417    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:24.799047 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.799017    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"]
Nov 01 06:09:24.800718 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:09:24.800689    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17e1eaf_fae0_49af_bfe9_8fe6b04bc163.slice/crio-1fb5c71e5103f65b88a0881672768aedd552cd6f4100a299545ba0044629c1f2 WatchSource:0}: Error finding container 1fb5c71e5103f65b88a0881672768aedd552cd6f4100a299545ba0044629c1f2: Status 404 returned error can't find the container with id 1fb5c71e5103f65b88a0881672768aedd552cd6f4100a299545ba0044629c1f2
Nov 01 06:09:24.803111 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.803078    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:09:24.803210 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.803141    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:09:24.803210 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:24.803169    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:09:25.097248 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:25.097215    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9" event={"ID":"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163","Type":"ContainerStarted","Data":"5aeb4125a45b0f389c81cc59878c22a41cef1e64aab13fa696d7240abe4802bd"}
Nov 01 06:09:25.097424 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:25.097273    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9" event={"ID":"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163","Type":"ContainerStarted","Data":"1fb5c71e5103f65b88a0881672768aedd552cd6f4100a299545ba0044629c1f2"}
Nov 01 06:09:25.222043 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:25.221989    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9" podStartSLOduration=2.221975364 podStartE2EDuration="2.221975364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:25.219640414 +0000 UTC m=+1402.430463925" watchObservedRunningTime="2026-11-01 06:09:25.221975364 +0000 UTC m=+1402.432798872"
Nov 01 06:09:25.364701 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:25.364609    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:25.366375 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:25.366317    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pskp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 06:09:25.366656 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:25.366623    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9" podUID="e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 06:09:25.932900 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:09:25.932861    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:09:26.367347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:26.365673    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pskp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 06:09:26.367347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:26.365739    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9" podUID="e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 06:09:27.369349 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:27.366679    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pskp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 06:09:27.369349 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:27.366745    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9" podUID="e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 06:09:27.691401 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:27.688487    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:28.363693 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:28.363659    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pskp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 06:09:28.363974 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:28.363947    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9" podUID="e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 06:09:28.622883 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:28.622790    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"]
Nov 01 06:09:28.709533 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:28.709498    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:29.367106 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:29.366654    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pskp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 06:09:29.367106 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:29.366713    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9" podUID="e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 06:09:30.365797 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:30.365757    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pskp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 06:09:30.366433 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:30.366396    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9" podUID="e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 06:09:31.363081 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:31.363039    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pskp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 06:09:31.363268 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:31.363111    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9" podUID="e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 06:09:32.362821 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:32.362786    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pskp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 01 06:09:32.363187 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:32.362836    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9" podUID="e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 01 06:09:32.460310 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:32.460262    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:34.405507 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:34.405464    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pskp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:09:34.405985 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:34.405529    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9" podUID="e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:09:35.414589 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:35.414551    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pskp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:09:35.415063 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:35.414613    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9" podUID="e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:09:35.426299 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:35.426264    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:35.426769 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:35.426748    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:35.427706 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:35.427682    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:09:37.364519 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:37.364483    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:42.304780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:42.304744    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:43.643037 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.642976    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq" podUID="1c40af1b-82e2-4a63-a08b-b371bc511bb2" containerName="main" containerID="cri-o://7b2edc0e1c26f9acf145688b07dcd6ebaa304412ddb135f5e2bedf795b7a93cc" gracePeriod=45
Nov 01 06:09:43.708299 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.708267    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"]
Nov 01 06:09:43.712131 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.712107    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"
Nov 01 06:09:43.748885 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.748853    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmvwt\" (UniqueName: \"kubernetes.io/projected/5593ec73-2b3f-4d24-b3f9-e142194caff4-kube-api-access-cmvwt\") pod \"maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c\" (UID: \"5593ec73-2b3f-4d24-b3f9-e142194caff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"
Nov 01 06:09:43.749034 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.748936    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5593ec73-2b3f-4d24-b3f9-e142194caff4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c\" (UID: \"5593ec73-2b3f-4d24-b3f9-e142194caff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"
Nov 01 06:09:43.749034 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.748984    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5593ec73-2b3f-4d24-b3f9-e142194caff4-metrics-certs\") pod \"maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c\" (UID: \"5593ec73-2b3f-4d24-b3f9-e142194caff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"
Nov 01 06:09:43.749164 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.749137    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5593ec73-2b3f-4d24-b3f9-e142194caff4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c\" (UID: \"5593ec73-2b3f-4d24-b3f9-e142194caff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"
Nov 01 06:09:43.765360 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.765336    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-8f45a2-serving-cert\""
Nov 01 06:09:43.819168 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.819135    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"]
Nov 01 06:09:43.849912 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.849876    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5593ec73-2b3f-4d24-b3f9-e142194caff4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c\" (UID: \"5593ec73-2b3f-4d24-b3f9-e142194caff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"
Nov 01 06:09:43.850068 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.850003    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmvwt\" (UniqueName: \"kubernetes.io/projected/5593ec73-2b3f-4d24-b3f9-e142194caff4-kube-api-access-cmvwt\") pod \"maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c\" (UID: \"5593ec73-2b3f-4d24-b3f9-e142194caff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"
Nov 01 06:09:43.850156 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.850134    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5593ec73-2b3f-4d24-b3f9-e142194caff4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c\" (UID: \"5593ec73-2b3f-4d24-b3f9-e142194caff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"
Nov 01 06:09:43.850213 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.850191    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5593ec73-2b3f-4d24-b3f9-e142194caff4-metrics-certs\") pod \"maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c\" (UID: \"5593ec73-2b3f-4d24-b3f9-e142194caff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"
Nov 01 06:09:43.850354 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:09:43.850295    5619 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-serving-cert: secret "maas-api-e2e-extmod-a-8f45a2-serving-cert" not found
Nov 01 06:09:43.850432 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:09:43.850418    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5593ec73-2b3f-4d24-b3f9-e142194caff4-maas-api-tls podName:5593ec73-2b3f-4d24-b3f9-e142194caff4 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:44.350392463 +0000 UTC m=+1421.561215970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5593ec73-2b3f-4d24-b3f9-e142194caff4-maas-api-tls") pod "maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c" (UID: "5593ec73-2b3f-4d24-b3f9-e142194caff4") : secret "maas-api-e2e-extmod-a-8f45a2-serving-cert" not found
Nov 01 06:09:43.850715 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.850698    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5593ec73-2b3f-4d24-b3f9-e142194caff4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c\" (UID: \"5593ec73-2b3f-4d24-b3f9-e142194caff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"
Nov 01 06:09:43.853145 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.853120    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5593ec73-2b3f-4d24-b3f9-e142194caff4-metrics-certs\") pod \"maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c\" (UID: \"5593ec73-2b3f-4d24-b3f9-e142194caff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"
Nov 01 06:09:43.893687 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.893616    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:43.912763 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.912740    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:09:43.913983 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.913958    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmvwt\" (UniqueName: \"kubernetes.io/projected/5593ec73-2b3f-4d24-b3f9-e142194caff4-kube-api-access-cmvwt\") pod \"maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c\" (UID: \"5593ec73-2b3f-4d24-b3f9-e142194caff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"
Nov 01 06:09:43.952096 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.951783    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-tmp-dir\") pod \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") "
Nov 01 06:09:43.952355 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.952259    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c40af1b-82e2-4a63-a08b-b371bc511bb2-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c40af1b-82e2-4a63-a08b-b371bc511bb2-tls-certs\") pod \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") "
Nov 01 06:09:43.956539 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.952396    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-dshm\") pod \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") "
Nov 01 06:09:43.956539 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.952414    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-tmp-dir" pod "1c40af1b-82e2-4a63-a08b-b371bc511bb2" (UID: "1c40af1b-82e2-4a63-a08b-b371bc511bb2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:09:43.956539 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.952429    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-home\" (UniqueName: \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-home\") pod \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") "
Nov 01 06:09:43.956539 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.952464    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-model-cache\") pod \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") "
Nov 01 06:09:43.956539 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.952550    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c40af1b-82e2-4a63-a08b-b371bc511bb2-kube-api-access-blkkp\" (UniqueName: \"kubernetes.io/projected/1c40af1b-82e2-4a63-a08b-b371bc511bb2-kube-api-access-blkkp\") pod \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\" (UID: \"1c40af1b-82e2-4a63-a08b-b371bc511bb2\") "
Nov 01 06:09:43.956539 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.953082    5619 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-tmp-dir\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:09:43.956539 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.954387    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-home" pod "1c40af1b-82e2-4a63-a08b-b371bc511bb2" (UID: "1c40af1b-82e2-4a63-a08b-b371bc511bb2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:09:43.956539 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.954578    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-model-cache" pod "1c40af1b-82e2-4a63-a08b-b371bc511bb2" (UID: "1c40af1b-82e2-4a63-a08b-b371bc511bb2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:09:43.956539 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.955750    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c40af1b-82e2-4a63-a08b-b371bc511bb2-tls-certs" pod "1c40af1b-82e2-4a63-a08b-b371bc511bb2" (UID: "1c40af1b-82e2-4a63-a08b-b371bc511bb2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:09:43.956539 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.955855    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c40af1b-82e2-4a63-a08b-b371bc511bb2-kube-api-access-blkkp" pod "1c40af1b-82e2-4a63-a08b-b371bc511bb2" (UID: "1c40af1b-82e2-4a63-a08b-b371bc511bb2"). InnerVolumeSpecName "kube-api-access-blkkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:09:43.958721 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:43.958699    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-dshm" pod "1c40af1b-82e2-4a63-a08b-b371bc511bb2" (UID: "1c40af1b-82e2-4a63-a08b-b371bc511bb2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:09:44.054017 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:44.053988    5619 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c40af1b-82e2-4a63-a08b-b371bc511bb2-tls-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:09:44.054017 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:44.054015    5619 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-dshm\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:09:44.054017 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:44.054024    5619 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-home\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:09:44.054248 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:44.054035    5619 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c40af1b-82e2-4a63-a08b-b371bc511bb2-model-cache\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:09:44.054248 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:44.054046    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blkkp\" (UniqueName: \"kubernetes.io/projected/1c40af1b-82e2-4a63-a08b-b371bc511bb2-kube-api-access-blkkp\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:09:44.083524 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:44.083490    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:44.220747 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:44.220653    5619 generic.go:356] "Generic (PLEG): container finished" podID="1c40af1b-82e2-4a63-a08b-b371bc511bb2" containerID="7b2edc0e1c26f9acf145688b07dcd6ebaa304412ddb135f5e2bedf795b7a93cc" exitCode=0
Nov 01 06:09:44.220747 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:44.220704    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"
Nov 01 06:09:44.220747 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:44.220710    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq" event={"ID":"1c40af1b-82e2-4a63-a08b-b371bc511bb2","Type":"ContainerDied","Data":"7b2edc0e1c26f9acf145688b07dcd6ebaa304412ddb135f5e2bedf795b7a93cc"}
Nov 01 06:09:44.220747 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:44.220748    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq" event={"ID":"1c40af1b-82e2-4a63-a08b-b371bc511bb2","Type":"ContainerDied","Data":"6f2768cd575e0ca19838fcf124a0a5456e6c132801d077f092bf1ec8f0b34e45"}
Nov 01 06:09:44.221054 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:44.220771    5619 scope.go:122] "RemoveContainer" containerID="7b2edc0e1c26f9acf145688b07dcd6ebaa304412ddb135f5e2bedf795b7a93cc"
Nov 01 06:09:44.230414 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:44.230390    5619 scope.go:122] "RemoveContainer" containerID="7b2edc0e1c26f9acf145688b07dcd6ebaa304412ddb135f5e2bedf795b7a93cc"
Nov 01 06:09:44.230678 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:09:44.230654    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b2edc0e1c26f9acf145688b07dcd6ebaa304412ddb135f5e2bedf795b7a93cc\": container with ID starting with 7b2edc0e1c26f9acf145688b07dcd6ebaa304412ddb135f5e2bedf795b7a93cc not found: ID does not exist" containerID="7b2edc0e1c26f9acf145688b07dcd6ebaa304412ddb135f5e2bedf795b7a93cc"
Nov 01 06:09:44.230734 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:44.230692    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b2edc0e1c26f9acf145688b07dcd6ebaa304412ddb135f5e2bedf795b7a93cc"} err="failed to get container status \"7b2edc0e1c26f9acf145688b07dcd6ebaa304412ddb135f5e2bedf795b7a93cc\": rpc error: code = NotFound desc = could not find container \"7b2edc0e1c26f9acf145688b07dcd6ebaa304412ddb135f5e2bedf795b7a93cc\": container with ID starting with 7b2edc0e1c26f9acf145688b07dcd6ebaa304412ddb135f5e2bedf795b7a93cc not found: ID does not exist"
Nov 01 06:09:44.357590 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:44.357558    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5593ec73-2b3f-4d24-b3f9-e142194caff4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c\" (UID: \"5593ec73-2b3f-4d24-b3f9-e142194caff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"
Nov 01 06:09:44.359676 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:44.359653    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5593ec73-2b3f-4d24-b3f9-e142194caff4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c\" (UID: \"5593ec73-2b3f-4d24-b3f9-e142194caff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"
Nov 01 06:09:44.388380 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:44.388357    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"]
Nov 01 06:09:44.392676 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:44.392655    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-58d523/ipp-route-58d523-kserve-75f8cf4b45-9zfmq"]
Nov 01 06:09:44.629233 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:44.629202    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"
Nov 01 06:09:44.964147 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:44.964070    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c40af1b-82e2-4a63-a08b-b371bc511bb2" path="/var/lib/kubelet/pods/1c40af1b-82e2-4a63-a08b-b371bc511bb2/volumes"
Nov 01 06:09:45.056200 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:45.056172    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"]
Nov 01 06:09:45.057739 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:09:45.057716    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5593ec73_2b3f_4d24_b3f9_e142194caff4.slice/crio-622c43c668ba5081f93a3950d53945af3c9c1fde3e0d4f674f9324eedc62ac3c WatchSource:0}: Error finding container 622c43c668ba5081f93a3950d53945af3c9c1fde3e0d4f674f9324eedc62ac3c: Status 404 returned error can't find the container with id 622c43c668ba5081f93a3950d53945af3c9c1fde3e0d4f674f9324eedc62ac3c
Nov 01 06:09:45.059445 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:45.059426    5619 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 06:09:45.228382 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:45.228288    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c" event={"ID":"5593ec73-2b3f-4d24-b3f9-e142194caff4","Type":"ContainerStarted","Data":"622c43c668ba5081f93a3950d53945af3c9c1fde3e0d4f674f9324eedc62ac3c"}
Nov 01 06:09:46.235458 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:46.235364    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c" event={"ID":"5593ec73-2b3f-4d24-b3f9-e142194caff4","Type":"ContainerStarted","Data":"1eb4daca213f62618408aae054f84ae3ec3de39213ad17b63d8303843feb176b"}
Nov 01 06:09:46.235458 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:46.235434    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"
Nov 01 06:09:46.355358 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:46.355292    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c" podStartSLOduration=2.5111953849999997 podStartE2EDuration="3.355278816s" totalImagesPullingTime="844.083431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:46.354624439 +0000 UTC m=+1423.565447952" watchObservedRunningTime="2026-11-01 06:09:46.355278816 +0000 UTC m=+1423.566102326"
Nov 01 06:09:48.852758 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:48.852719    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:52.247350 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:52.247288    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"
Nov 01 06:09:53.666147 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:53.666098    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:54.062360 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.062257    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"]
Nov 01 06:09:54.062850 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.062834    5619 cpu_manager.go:414] "containerMap: removing container" podUID="1c40af1b-82e2-4a63-a08b-b371bc511bb2" containerName="main"
Nov 01 06:09:54.062892 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.062851    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c40af1b-82e2-4a63-a08b-b371bc511bb2" containerName="main"
Nov 01 06:09:54.062980 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.062971    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c40af1b-82e2-4a63-a08b-b371bc511bb2" containerName="main"
Nov 01 06:09:54.066445 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.066420    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.187074 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.187035    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.187379 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.187351    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.187558 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.187538    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p64nr\" (UniqueName: \"kubernetes.io/projected/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-kube-api-access-p64nr\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.187719 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.187701    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.187859 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.187842    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.188074 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.188045    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.188194 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.188176    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.188265 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.188251    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.188364 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.188275    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.273288 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.273257    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"]
Nov 01 06:09:54.290062 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.290026    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p64nr\" (UniqueName: \"kubernetes.io/projected/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-kube-api-access-p64nr\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.290245 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.290106    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.290245 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.290155    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.290245 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.290195    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.290463 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.290255    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.290463 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.290344    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.290463 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.290375    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.290463 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.290458    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.290673 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.290518    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.290807 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.290784    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.290881 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.290859    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.291729 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.291022    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.291729 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.291592    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.291729 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.291679    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.293548 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.293506    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.297410 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.297385    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.352075 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.352032    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.352238 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.352215    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p64nr\" (UniqueName: \"kubernetes.io/projected/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-kube-api-access-p64nr\") pod \"maas-default-gateway-openshift-default-855f9574cf-6xdr7\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.383859 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.383832    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:54.724399 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.724289    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:54.816183 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.816147    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"]
Nov 01 06:09:54.817470 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:09:54.817444    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231e12b3_f2c4_4633_8a9b_4dcfc8c34206.slice/crio-f79b04ac50504f852e4744fe02e9a47cf2babdf63719418e172d5809fc1f5414 WatchSource:0}: Error finding container f79b04ac50504f852e4744fe02e9a47cf2babdf63719418e172d5809fc1f5414: Status 404 returned error can't find the container with id f79b04ac50504f852e4744fe02e9a47cf2babdf63719418e172d5809fc1f5414
Nov 01 06:09:54.819821 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.819790    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:09:54.819888 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.819852    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:09:54.819888 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.819880    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:09:54.863693 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:54.863660    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:55.294289 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:55.294248    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7" event={"ID":"231e12b3-f2c4-4633-8a9b-4dcfc8c34206","Type":"ContainerStarted","Data":"6e307a6f756f4ca43804d8d82e122eca5705ab8619e430f43d4cac8d515e6331"}
Nov 01 06:09:55.294616 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:55.294316    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7" event={"ID":"231e12b3-f2c4-4633-8a9b-4dcfc8c34206","Type":"ContainerStarted","Data":"f79b04ac50504f852e4744fe02e9a47cf2babdf63719418e172d5809fc1f5414"}
Nov 01 06:09:55.384650 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:55.384620    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:09:55.386556 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:55.386528    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6xdr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 01 06:09:55.386706 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:55.386576    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7" podUID="231e12b3-f2c4-4633-8a9b-4dcfc8c34206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 01 06:09:55.494669 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:55.494617    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7" podStartSLOduration=1.494602746 podStartE2EDuration="1.494602746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:55.466049541 +0000 UTC m=+1432.676873053" watchObservedRunningTime="2026-11-01 06:09:55.494602746 +0000 UTC m=+1432.705426255"
Nov 01 06:09:55.905867 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:09:55.905825    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:09:56.384665 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:56.384625    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6xdr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 01 06:09:56.384882 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:56.384703    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7" podUID="231e12b3-f2c4-4633-8a9b-4dcfc8c34206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 01 06:09:57.384839 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:57.384804    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6xdr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 01 06:09:57.385247 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:57.384880    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7" podUID="231e12b3-f2c4-4633-8a9b-4dcfc8c34206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 01 06:09:58.392347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:58.391780    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6xdr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 01 06:09:58.392347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:58.391858    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7" podUID="231e12b3-f2c4-4633-8a9b-4dcfc8c34206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 01 06:09:58.511758 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:58.511718    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:09:59.394018 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:59.393566    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6xdr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 01 06:09:59.394018 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:09:59.393627    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7" podUID="231e12b3-f2c4-4633-8a9b-4dcfc8c34206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 01 06:10:00.385345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:00.384487    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6xdr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 01 06:10:00.385345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:00.384576    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7" podUID="231e12b3-f2c4-4633-8a9b-4dcfc8c34206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 01 06:10:01.384746 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:01.384703    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6xdr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 01 06:10:01.385267 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:01.384804    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7" podUID="231e12b3-f2c4-4633-8a9b-4dcfc8c34206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 01 06:10:02.385345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:02.384775    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6xdr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 01 06:10:02.385345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:02.384854    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7" podUID="231e12b3-f2c4-4633-8a9b-4dcfc8c34206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 01 06:10:03.503011 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:03.502909    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:03.644903 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:03.644866    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:03.932119 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:03.930512    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:10:03.932638 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:03.932434    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:10:03.933718 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:03.933696    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:10:04.310908 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.310816    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"]
Nov 01 06:10:04.317009 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.316978    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.381205 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.381169    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"]
Nov 01 06:10:04.402899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.402866    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:04.413444 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.413411    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-token\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.413606 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.413475    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-envoy\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.413606 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.413560    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-data\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.413711 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.413609    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ae3e00-d79f-473e-ae51-c5b5029f5353-istiod-ca-cert\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.413768 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.413723    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-credential-socket\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.413824 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.413795    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-workload-certs\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.413824 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.413813    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtmqw\" (UniqueName: \"kubernetes.io/projected/39ae3e00-d79f-473e-ae51-c5b5029f5353-kube-api-access-qtmqw\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.413959 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.413942    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-workload-socket\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.414019 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.413970    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-podinfo\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.443574 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.443554    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-ec5596-openshift-default-dockercfg-pncn7\""
Nov 01 06:10:04.514919 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.514883    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-credential-socket\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.515274 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.514983    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-workload-certs\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.515274 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.515004    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtmqw\" (UniqueName: \"kubernetes.io/projected/39ae3e00-d79f-473e-ae51-c5b5029f5353-kube-api-access-qtmqw\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.515274 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.515136    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-workload-socket\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.515274 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.515170    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-podinfo\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.515274 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.515238    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-token\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.515570 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.515385    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-envoy\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.515570 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.515431    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-data\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.515570 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.515440    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-workload-certs\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.515570 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.515387    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-credential-socket\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.515570 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.515540    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ae3e00-d79f-473e-ae51-c5b5029f5353-istiod-ca-cert\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.515827 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.515651    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-workload-socket\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.516043 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.516015    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-data\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.516449 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.516426    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ae3e00-d79f-473e-ae51-c5b5029f5353-istiod-ca-cert\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.517917 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.517897    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-podinfo\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.518160 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.518141    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-envoy\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.593989 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.593955    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-token\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.623687 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.616118    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtmqw\" (UniqueName: \"kubernetes.io/projected/39ae3e00-d79f-473e-ae51-c5b5029f5353-kube-api-access-qtmqw\") pod \"e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") " pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:04.623687 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.623285    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:04.632578 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:04.632528    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:05.077708 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:05.077675    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"]
Nov 01 06:10:05.079026 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:10:05.079000    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ae3e00_d79f_473e_ae51_c5b5029f5353.slice/crio-8e3e4de014edfe309bf5478d0247908bc759f482e6cd78dcb1b2385f10109e64 WatchSource:0}: Error finding container 8e3e4de014edfe309bf5478d0247908bc759f482e6cd78dcb1b2385f10109e64: Status 404 returned error can't find the container with id 8e3e4de014edfe309bf5478d0247908bc759f482e6cd78dcb1b2385f10109e64
Nov 01 06:10:05.082253 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:05.082222    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:10:05.082393 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:05.082285    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:10:05.082393 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:05.082313    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:10:05.427240 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:05.427085    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv" event={"ID":"39ae3e00-d79f-473e-ae51-c5b5029f5353","Type":"ContainerStarted","Data":"5e37f4871a4556972d0d93b79f58144305d7bbe0a03343b70e755935605b7b34"}
Nov 01 06:10:05.427240 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:05.427192    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv" event={"ID":"39ae3e00-d79f-473e-ae51-c5b5029f5353","Type":"ContainerStarted","Data":"8e3e4de014edfe309bf5478d0247908bc759f482e6cd78dcb1b2385f10109e64"}
Nov 01 06:10:05.562373 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:05.559875    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv" podStartSLOduration=1.5598565899999999 podStartE2EDuration="1.55985659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:05.558768621 +0000 UTC m=+1442.769592131" watchObservedRunningTime="2026-11-01 06:10:05.55985659 +0000 UTC m=+1442.770680098"
Nov 01 06:10:05.632714 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:05.632664    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:05.720797 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:05.720696    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:06.423780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:06.423739    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:06.638371 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:06.638340    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:07.441237 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:07.441203    5619 generic.go:356] "Generic (PLEG): container finished" podID="4482833c-590d-44a1-a485-651970b9abf6" containerID="4c6426a84a6077a5ae7e36ce85a8f77889a878a31b88eb5c96419fa2a96db924" exitCode=0
Nov 01 06:10:07.441480 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:07.441275    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl" event={"ID":"4482833c-590d-44a1-a485-651970b9abf6","Type":"ContainerDied","Data":"4c6426a84a6077a5ae7e36ce85a8f77889a878a31b88eb5c96419fa2a96db924"}
Nov 01 06:10:07.441567 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:07.441530    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:07.441849 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:07.441832    5619 scope.go:122] "RemoveContainer" containerID="4c6426a84a6077a5ae7e36ce85a8f77889a878a31b88eb5c96419fa2a96db924"
Nov 01 06:10:07.442732 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:07.442577    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:10:07.443243 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:07.443220    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:10:07.443298 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:07.443284    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:10:07.443357 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:07.443341    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:10:08.455357 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:08.450902    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl" event={"ID":"4482833c-590d-44a1-a485-651970b9abf6","Type":"ContainerStarted","Data":"a7b937f3a550f7456b7820648e64c82ff01f10c7cdde356876e658bc9863af06"}
Nov 01 06:10:08.674347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:08.668372    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:08.825381 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:08.825285    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:10:08.825558 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:08.825534    5619 patch_prober.go:29] interesting pod/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 01 06:10:08.825634 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:08.825616    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl" podUID="4482833c-590d-44a1-a485-651970b9abf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 01 06:10:09.825526 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:09.825488    5619 patch_prober.go:29] interesting pod/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 01 06:10:09.826028 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:09.825540    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl" podUID="4482833c-590d-44a1-a485-651970b9abf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 01 06:10:10.825398 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:10.825291    5619 patch_prober.go:29] interesting pod/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 01 06:10:10.825398 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:10.825383    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl" podUID="4482833c-590d-44a1-a485-651970b9abf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 01 06:10:11.825576 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:11.825545    5619 patch_prober.go:29] interesting pod/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 01 06:10:11.825973 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:11.825628    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl" podUID="4482833c-590d-44a1-a485-651970b9abf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 01 06:10:12.181933 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:12.181894    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:12.825998 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:12.825954    5619 patch_prober.go:29] interesting pod/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 01 06:10:12.826591 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:12.826015    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl" podUID="4482833c-590d-44a1-a485-651970b9abf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 01 06:10:13.502920 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:13.501954    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:13.830052 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:13.829971    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:10:13.830543 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:13.830364    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:10:13.831038 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:13.831017    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:10:18.466399 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:18.466365    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:18.663614 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:18.663577    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:18.918229 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:18.918183    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:19.584774 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:19.584727    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:21.252647 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:21.252600    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:21.650903 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:21.650871    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:21.825248 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:21.825202    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:23.840027 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:23.839989    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:25.881454 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:10:25.881411    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:10:26.597100 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:26.597060    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:26.744201 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:26.744158    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:30.354561 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:30.354520    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:30.548621 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:30.548590    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"]
Nov 01 06:10:30.552712 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:30.552694    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"
Nov 01 06:10:30.614478 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:30.614401    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"]
Nov 01 06:10:30.659032 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:30.658993    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"agc-cred-def9d068021f\""
Nov 01 06:10:30.690777 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:30.690740    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d987b060-9d89-4135-8f0a-dd5a8c1afc43-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd\" (UID: \"d987b060-9d89-4135-8f0a-dd5a8c1afc43\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"
Nov 01 06:10:30.690935 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:30.690910    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d987b060-9d89-4135-8f0a-dd5a8c1afc43-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd\" (UID: \"d987b060-9d89-4135-8f0a-dd5a8c1afc43\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"
Nov 01 06:10:30.690982 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:30.690941    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9brjd\" (UniqueName: \"kubernetes.io/projected/d987b060-9d89-4135-8f0a-dd5a8c1afc43-kube-api-access-9brjd\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd\" (UID: \"d987b060-9d89-4135-8f0a-dd5a8c1afc43\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"
Nov 01 06:10:30.691056 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:30.691041    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d987b060-9d89-4135-8f0a-dd5a8c1afc43-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd\" (UID: \"d987b060-9d89-4135-8f0a-dd5a8c1afc43\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"
Nov 01 06:10:30.792170 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:30.792135    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d987b060-9d89-4135-8f0a-dd5a8c1afc43-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd\" (UID: \"d987b060-9d89-4135-8f0a-dd5a8c1afc43\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"
Nov 01 06:10:30.792368 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:30.792178    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9brjd\" (UniqueName: \"kubernetes.io/projected/d987b060-9d89-4135-8f0a-dd5a8c1afc43-kube-api-access-9brjd\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd\" (UID: \"d987b060-9d89-4135-8f0a-dd5a8c1afc43\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"
Nov 01 06:10:30.792368 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:30.792255    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d987b060-9d89-4135-8f0a-dd5a8c1afc43-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd\" (UID: \"d987b060-9d89-4135-8f0a-dd5a8c1afc43\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"
Nov 01 06:10:30.792505 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:30.792414    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d987b060-9d89-4135-8f0a-dd5a8c1afc43-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd\" (UID: \"d987b060-9d89-4135-8f0a-dd5a8c1afc43\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"
Nov 01 06:10:30.792938 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:30.792908    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d987b060-9d89-4135-8f0a-dd5a8c1afc43-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd\" (UID: \"d987b060-9d89-4135-8f0a-dd5a8c1afc43\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"
Nov 01 06:10:30.793065 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:30.793029    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d987b060-9d89-4135-8f0a-dd5a8c1afc43-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd\" (UID: \"d987b060-9d89-4135-8f0a-dd5a8c1afc43\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"
Nov 01 06:10:30.794963 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:30.794941    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d987b060-9d89-4135-8f0a-dd5a8c1afc43-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd\" (UID: \"d987b060-9d89-4135-8f0a-dd5a8c1afc43\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"
Nov 01 06:10:30.853849 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:30.853798    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9brjd\" (UniqueName: \"kubernetes.io/projected/d987b060-9d89-4135-8f0a-dd5a8c1afc43-kube-api-access-9brjd\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd\" (UID: \"d987b060-9d89-4135-8f0a-dd5a8c1afc43\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"
Nov 01 06:10:30.867530 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:30.867475    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"
Nov 01 06:10:31.245665 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:31.245638    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"]
Nov 01 06:10:31.247369 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:10:31.247300    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd987b060_9d89_4135_8f0a_dd5a8c1afc43.slice/crio-62023575b2a422688a1e6f6de11a6c04e6290df7a6aa62aa72fd84f5f5c8bf3a WatchSource:0}: Error finding container 62023575b2a422688a1e6f6de11a6c04e6290df7a6aa62aa72fd84f5f5c8bf3a: Status 404 returned error can't find the container with id 62023575b2a422688a1e6f6de11a6c04e6290df7a6aa62aa72fd84f5f5c8bf3a
Nov 01 06:10:31.622646 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:31.622611    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd" event={"ID":"d987b060-9d89-4135-8f0a-dd5a8c1afc43","Type":"ContainerStarted","Data":"c4245198ca4c7e839497adcf7190422479f44c89fa4d22939d02a328320238a1"}
Nov 01 06:10:31.623043 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:31.622669    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"
Nov 01 06:10:31.623043 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:31.622687    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd" event={"ID":"d987b060-9d89-4135-8f0a-dd5a8c1afc43","Type":"ContainerStarted","Data":"62023575b2a422688a1e6f6de11a6c04e6290df7a6aa62aa72fd84f5f5c8bf3a"}
Nov 01 06:10:31.742824 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:31.742772    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd" podStartSLOduration=1.742758166 podStartE2EDuration="1.742758166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:31.740730828 +0000 UTC m=+1468.951554338" watchObservedRunningTime="2026-11-01 06:10:31.742758166 +0000 UTC m=+1468.953581675"
Nov 01 06:10:33.198727 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:33.198691    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:34.110562 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:34.110527    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:34.304100 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:34.304060    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:35.270438 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:35.270395    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:35.928240 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:35.927948    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:36.073080 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:36.073033    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:38.076355 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:38.076305    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"]
Nov 01 06:10:38.079602 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:38.079581    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"
Nov 01 06:10:39.297464 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:39.297425    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:39.711674 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:39.711638    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:40.484596 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:40.484559    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:40.715006 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:40.714971    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:40.861879 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:40.861848    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:42.103838 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:42.103799    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:42.762060 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:42.761914    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:42.917228 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:42.917196    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:43.062994 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.062902    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:43.077004 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.076950    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd" podUID="d987b060-9d89-4135-8f0a-dd5a8c1afc43" containerName="payload-processing" containerID="cri-o://c4245198ca4c7e839497adcf7190422479f44c89fa4d22939d02a328320238a1" gracePeriod=25
Nov 01 06:10:43.276013 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.266841    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:43.351028 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.351005    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"
Nov 01 06:10:43.383679 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.383650    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d987b060-9d89-4135-8f0a-dd5a8c1afc43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d987b060-9d89-4135-8f0a-dd5a8c1afc43-plugins-config-volume\") pod \"d987b060-9d89-4135-8f0a-dd5a8c1afc43\" (UID: \"d987b060-9d89-4135-8f0a-dd5a8c1afc43\") "
Nov 01 06:10:43.383824 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.383727    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d987b060-9d89-4135-8f0a-dd5a8c1afc43-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d987b060-9d89-4135-8f0a-dd5a8c1afc43-provider-credentials\") pod \"d987b060-9d89-4135-8f0a-dd5a8c1afc43\" (UID: \"d987b060-9d89-4135-8f0a-dd5a8c1afc43\") "
Nov 01 06:10:43.383882 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.383819    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d987b060-9d89-4135-8f0a-dd5a8c1afc43-kube-api-access-9brjd\" (UniqueName: \"kubernetes.io/projected/d987b060-9d89-4135-8f0a-dd5a8c1afc43-kube-api-access-9brjd\") pod \"d987b060-9d89-4135-8f0a-dd5a8c1afc43\" (UID: \"d987b060-9d89-4135-8f0a-dd5a8c1afc43\") "
Nov 01 06:10:43.383942 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.383931    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d987b060-9d89-4135-8f0a-dd5a8c1afc43-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d987b060-9d89-4135-8f0a-dd5a8c1afc43-routing-overlay-volume\") pod \"d987b060-9d89-4135-8f0a-dd5a8c1afc43\" (UID: \"d987b060-9d89-4135-8f0a-dd5a8c1afc43\") "
Nov 01 06:10:43.384096 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.384067    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d987b060-9d89-4135-8f0a-dd5a8c1afc43-plugins-config-volume" pod "d987b060-9d89-4135-8f0a-dd5a8c1afc43" (UID: "d987b060-9d89-4135-8f0a-dd5a8c1afc43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:10:43.384364 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.384316    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d987b060-9d89-4135-8f0a-dd5a8c1afc43-routing-overlay-volume" pod "d987b060-9d89-4135-8f0a-dd5a8c1afc43" (UID: "d987b060-9d89-4135-8f0a-dd5a8c1afc43"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:10:43.384479 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.384458    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d987b060-9d89-4135-8f0a-dd5a8c1afc43-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:43.384537 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.384488    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d987b060-9d89-4135-8f0a-dd5a8c1afc43-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:43.385839 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.385816    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d987b060-9d89-4135-8f0a-dd5a8c1afc43-provider-credentials" pod "d987b060-9d89-4135-8f0a-dd5a8c1afc43" (UID: "d987b060-9d89-4135-8f0a-dd5a8c1afc43"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:10:43.385921 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.385888    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d987b060-9d89-4135-8f0a-dd5a8c1afc43-kube-api-access-9brjd" pod "d987b060-9d89-4135-8f0a-dd5a8c1afc43" (UID: "d987b060-9d89-4135-8f0a-dd5a8c1afc43"). InnerVolumeSpecName "kube-api-access-9brjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:10:43.486443 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.486406    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d987b060-9d89-4135-8f0a-dd5a8c1afc43-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:43.486443 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.486440    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9brjd\" (UniqueName: \"kubernetes.io/projected/d987b060-9d89-4135-8f0a-dd5a8c1afc43-kube-api-access-9brjd\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:10:43.777300 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.777264    5619 generic.go:356] "Generic (PLEG): container finished" podID="d987b060-9d89-4135-8f0a-dd5a8c1afc43" containerID="c4245198ca4c7e839497adcf7190422479f44c89fa4d22939d02a328320238a1" exitCode=0
Nov 01 06:10:43.777457 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.777311    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"
Nov 01 06:10:43.777457 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.777311    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd" event={"ID":"d987b060-9d89-4135-8f0a-dd5a8c1afc43","Type":"ContainerDied","Data":"c4245198ca4c7e839497adcf7190422479f44c89fa4d22939d02a328320238a1"}
Nov 01 06:10:43.777457 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.777441    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd" event={"ID":"d987b060-9d89-4135-8f0a-dd5a8c1afc43","Type":"ContainerDied","Data":"62023575b2a422688a1e6f6de11a6c04e6290df7a6aa62aa72fd84f5f5c8bf3a"}
Nov 01 06:10:43.777592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.777471    5619 scope.go:122] "RemoveContainer" containerID="c4245198ca4c7e839497adcf7190422479f44c89fa4d22939d02a328320238a1"
Nov 01 06:10:43.787857 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.787835    5619 scope.go:122] "RemoveContainer" containerID="c4245198ca4c7e839497adcf7190422479f44c89fa4d22939d02a328320238a1"
Nov 01 06:10:43.788153 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:10:43.788128    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4245198ca4c7e839497adcf7190422479f44c89fa4d22939d02a328320238a1\": container with ID starting with c4245198ca4c7e839497adcf7190422479f44c89fa4d22939d02a328320238a1 not found: ID does not exist" containerID="c4245198ca4c7e839497adcf7190422479f44c89fa4d22939d02a328320238a1"
Nov 01 06:10:43.788267 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.788162    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4245198ca4c7e839497adcf7190422479f44c89fa4d22939d02a328320238a1"} err="failed to get container status \"c4245198ca4c7e839497adcf7190422479f44c89fa4d22939d02a328320238a1\": rpc error: code = NotFound desc = could not find container \"c4245198ca4c7e839497adcf7190422479f44c89fa4d22939d02a328320238a1\": container with ID starting with c4245198ca4c7e839497adcf7190422479f44c89fa4d22939d02a328320238a1 not found: ID does not exist"
Nov 01 06:10:43.887031 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.886989    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:43.953009 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.952976    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"]
Nov 01 06:10:43.963506 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:43.963481    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f9872qgcd"]
Nov 01 06:10:44.964408 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:44.964374    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d987b060-9d89-4135-8f0a-dd5a8c1afc43" path="/var/lib/kubelet/pods/d987b060-9d89-4135-8f0a-dd5a8c1afc43/volumes"
Nov 01 06:10:45.523890 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:45.523849    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:46.566786 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:46.566741    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:46.777359 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:46.777300    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:47.930435 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:47.930389    5619 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal" podUID="ef0f634a7979920e9df606f3bb474aac" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:10:50.547254 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:50.547217    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:50.950065 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:50.950031    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:51.218028 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:51.217933    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:51.365290 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:51.365254    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c"]
Nov 01 06:10:51.365986 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:51.365966    5619 cpu_manager.go:414] "containerMap: removing container" podUID="d987b060-9d89-4135-8f0a-dd5a8c1afc43" containerName="payload-processing"
Nov 01 06:10:51.365986 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:51.365986    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d987b060-9d89-4135-8f0a-dd5a8c1afc43" containerName="payload-processing"
Nov 01 06:10:51.366188 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:51.366141    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="d987b060-9d89-4135-8f0a-dd5a8c1afc43" containerName="payload-processing"
Nov 01 06:10:51.369543 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:51.369518    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c"
Nov 01 06:10:51.423672 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:51.423635    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-7d25db\""
Nov 01 06:10:51.474685 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:51.474597    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-7d25db-dockercfg-gvvrq\""
Nov 01 06:10:51.478261 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:51.478231    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c"]
Nov 01 06:10:51.515343 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:51.499167    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk48d\" (UniqueName: \"kubernetes.io/projected/e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6-kube-api-access-nk48d\") pod \"payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c\" (UID: \"e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c"
Nov 01 06:10:51.515343 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:51.499248    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c\" (UID: \"e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c"
Nov 01 06:10:51.602469 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:51.602140    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk48d\" (UniqueName: \"kubernetes.io/projected/e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6-kube-api-access-nk48d\") pod \"payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c\" (UID: \"e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c"
Nov 01 06:10:51.602989 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:51.602683    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c\" (UID: \"e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c"
Nov 01 06:10:51.603622 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:51.603593    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c\" (UID: \"e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c"
Nov 01 06:10:51.664134 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:51.664104    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk48d\" (UniqueName: \"kubernetes.io/projected/e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6-kube-api-access-nk48d\") pod \"payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c\" (UID: \"e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c"
Nov 01 06:10:51.683523 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:51.683494    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c"
Nov 01 06:10:52.060307 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:52.060275    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c"]
Nov 01 06:10:52.061057 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:10:52.061029    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49d0f57_4d8e_45fc_9f7b_2d37e5d731c6.slice/crio-c9a0b6e8c7114f8978da2e6ad022a96f056634204fd1eb56d14154fb120ea266 WatchSource:0}: Error finding container c9a0b6e8c7114f8978da2e6ad022a96f056634204fd1eb56d14154fb120ea266: Status 404 returned error can't find the container with id c9a0b6e8c7114f8978da2e6ad022a96f056634204fd1eb56d14154fb120ea266
Nov 01 06:10:52.831423 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:52.831384    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c" event={"ID":"e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6","Type":"ContainerStarted","Data":"7086e5794e23a5387412c1ab877417d0aaad7b97eafbcbd00039d4af9ff51eb2"}
Nov 01 06:10:52.831798 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:52.831435    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c" event={"ID":"e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6","Type":"ContainerStarted","Data":"c9a0b6e8c7114f8978da2e6ad022a96f056634204fd1eb56d14154fb120ea266"}
Nov 01 06:10:52.831798 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:52.831466    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c"
Nov 01 06:10:52.955614 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:52.955547    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c" podStartSLOduration=1.955524691 podStartE2EDuration="1.955524691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:52.950618245 +0000 UTC m=+1490.161441757" watchObservedRunningTime="2026-11-01 06:10:52.955524691 +0000 UTC m=+1490.166348202"
Nov 01 06:10:55.060123 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:55.060086    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:55.258985 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:55.258945    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:55.494909 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:55.494873    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:55.827229 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:55.827137    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:55.888002 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:10:55.887949    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:10:55.981280 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:55.981252    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:56.607201 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:56.607171    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:56.754042 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:56.754006    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:59.281051 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:59.281017    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:59.475526 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:59.475493    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:10:59.990816 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:10:59.990771    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:00.789278 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:00.789241    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:03.668028 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:03.667998    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"]
Nov 01 06:11:03.673454 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:03.673434    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:03.728427 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:03.728399    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"agc-cred-c01d3031634c\""
Nov 01 06:11:03.764336 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:03.764281    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add83dd5-a128-499e-9e8b-00142cc7bc94-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq\" (UID: \"add83dd5-a128-499e-9e8b-00142cc7bc94\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:03.764543 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:03.764512    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-668j4\" (UniqueName: \"kubernetes.io/projected/add83dd5-a128-499e-9e8b-00142cc7bc94-kube-api-access-668j4\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq\" (UID: \"add83dd5-a128-499e-9e8b-00142cc7bc94\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:03.764681 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:03.764653    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/add83dd5-a128-499e-9e8b-00142cc7bc94-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq\" (UID: \"add83dd5-a128-499e-9e8b-00142cc7bc94\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:03.764813 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:03.764793    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/add83dd5-a128-499e-9e8b-00142cc7bc94-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq\" (UID: \"add83dd5-a128-499e-9e8b-00142cc7bc94\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:03.781359 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:03.781337    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"]
Nov 01 06:11:03.841789 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:03.841751    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c"
Nov 01 06:11:03.866496 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:03.866464    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-668j4\" (UniqueName: \"kubernetes.io/projected/add83dd5-a128-499e-9e8b-00142cc7bc94-kube-api-access-668j4\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq\" (UID: \"add83dd5-a128-499e-9e8b-00142cc7bc94\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:03.866651 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:03.866550    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/add83dd5-a128-499e-9e8b-00142cc7bc94-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq\" (UID: \"add83dd5-a128-499e-9e8b-00142cc7bc94\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:03.866651 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:03.866623    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/add83dd5-a128-499e-9e8b-00142cc7bc94-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq\" (UID: \"add83dd5-a128-499e-9e8b-00142cc7bc94\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:03.866734 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:03.866667    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add83dd5-a128-499e-9e8b-00142cc7bc94-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq\" (UID: \"add83dd5-a128-499e-9e8b-00142cc7bc94\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:03.867424 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:03.867401    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/add83dd5-a128-499e-9e8b-00142cc7bc94-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq\" (UID: \"add83dd5-a128-499e-9e8b-00142cc7bc94\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:03.867557 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:03.867511    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add83dd5-a128-499e-9e8b-00142cc7bc94-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq\" (UID: \"add83dd5-a128-499e-9e8b-00142cc7bc94\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:03.869300 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:03.869276    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/add83dd5-a128-499e-9e8b-00142cc7bc94-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq\" (UID: \"add83dd5-a128-499e-9e8b-00142cc7bc94\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:03.928658 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:03.928586    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-668j4\" (UniqueName: \"kubernetes.io/projected/add83dd5-a128-499e-9e8b-00142cc7bc94-kube-api-access-668j4\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq\" (UID: \"add83dd5-a128-499e-9e8b-00142cc7bc94\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:03.987781 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:03.987740    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:05.329793 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:05.329756    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"]
Nov 01 06:11:05.334819 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:05.334802    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:05.385789 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:05.385757    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cmqp\" (UniqueName: \"kubernetes.io/projected/d21d21dc-79f2-40c1-b701-e75e87bd144c-kube-api-access-6cmqp\") pod \"maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn\" (UID: \"d21d21dc-79f2-40c1-b701-e75e87bd144c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:05.385934 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:05.385858    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d21d21dc-79f2-40c1-b701-e75e87bd144c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn\" (UID: \"d21d21dc-79f2-40c1-b701-e75e87bd144c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:05.385991 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:05.385978    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d21d21dc-79f2-40c1-b701-e75e87bd144c-metrics-certs\") pod \"maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn\" (UID: \"d21d21dc-79f2-40c1-b701-e75e87bd144c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:05.386037 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:05.386024    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d21d21dc-79f2-40c1-b701-e75e87bd144c-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn\" (UID: \"d21d21dc-79f2-40c1-b701-e75e87bd144c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:05.389839 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:05.389816    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-ec5596-serving-cert\""
Nov 01 06:11:05.443585 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:05.443556    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"]
Nov 01 06:11:05.487667 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:05.487635    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cmqp\" (UniqueName: \"kubernetes.io/projected/d21d21dc-79f2-40c1-b701-e75e87bd144c-kube-api-access-6cmqp\") pod \"maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn\" (UID: \"d21d21dc-79f2-40c1-b701-e75e87bd144c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:05.487856 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:05.487702    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d21d21dc-79f2-40c1-b701-e75e87bd144c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn\" (UID: \"d21d21dc-79f2-40c1-b701-e75e87bd144c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:05.487856 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:05.487817    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d21d21dc-79f2-40c1-b701-e75e87bd144c-metrics-certs\") pod \"maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn\" (UID: \"d21d21dc-79f2-40c1-b701-e75e87bd144c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:05.487977 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:05.487848    5619 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-serving-cert: secret "maas-api-e2e-praxis-b-ec5596-serving-cert" not found
Nov 01 06:11:05.487977 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:05.487897    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d21d21dc-79f2-40c1-b701-e75e87bd144c-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn\" (UID: \"d21d21dc-79f2-40c1-b701-e75e87bd144c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:05.487977 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:05.487921    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d21d21dc-79f2-40c1-b701-e75e87bd144c-maas-api-tls podName:d21d21dc-79f2-40c1-b701-e75e87bd144c nodeName:}" failed. No retries permitted until 2026-11-01 06:11:05.987898234 +0000 UTC m=+1503.198721725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d21d21dc-79f2-40c1-b701-e75e87bd144c-maas-api-tls") pod "maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn" (UID: "d21d21dc-79f2-40c1-b701-e75e87bd144c") : secret "maas-api-e2e-praxis-b-ec5596-serving-cert" not found
Nov 01 06:11:05.488749 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:05.488724    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d21d21dc-79f2-40c1-b701-e75e87bd144c-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn\" (UID: \"d21d21dc-79f2-40c1-b701-e75e87bd144c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:05.490285 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:05.490264    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d21d21dc-79f2-40c1-b701-e75e87bd144c-metrics-certs\") pod \"maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn\" (UID: \"d21d21dc-79f2-40c1-b701-e75e87bd144c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:05.522753 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:05.522720    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:05.551740 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:05.551710    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cmqp\" (UniqueName: \"kubernetes.io/projected/d21d21dc-79f2-40c1-b701-e75e87bd144c-kube-api-access-6cmqp\") pod \"maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn\" (UID: \"d21d21dc-79f2-40c1-b701-e75e87bd144c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:05.682736 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:05.682690    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:05.830764 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:05.830721    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:05.998267 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:05.998175    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d21d21dc-79f2-40c1-b701-e75e87bd144c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn\" (UID: \"d21d21dc-79f2-40c1-b701-e75e87bd144c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:06.000640 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.000613    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d21d21dc-79f2-40c1-b701-e75e87bd144c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn\" (UID: \"d21d21dc-79f2-40c1-b701-e75e87bd144c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:06.249603 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.249505    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:06.551045 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.550963    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"]
Nov 01 06:11:06.560849 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.560825    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.591584 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:06.591552    5619 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 06:11:06.591584 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf_add83dd5-a128-499e-9e8b-00142cc7bc94_0(51534e5154139093b2f028913374ce5c67787f899d5a36728ae232589cd59703): error adding pod ai-tenant-e2e-extmod-b-3986bf_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"51534e5154139093b2f028913374ce5c67787f899d5a36728ae232589cd59703" Netns:"/var/run/netns/5f71eb12-93c5-4e14-9948-0604e0c5322f" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-extmod-b-3986bf;K8S_POD_NAME=payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq;K8S_POD_INFRA_CONTAINER_ID=51534e5154139093b2f028913374ce5c67787f899d5a36728ae232589cd59703;K8S_POD_UID=add83dd5-a128-499e-9e8b-00142cc7bc94" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq] networking: Multus: [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq/add83dd5-a128-499e-9e8b-00142cc7bc94]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-extmod-b-3986bf/pods/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq?timeout=1m0s": context deadline exceeded
Nov 01 06:11:06.591584 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:06.591584 ip-10-0-129-213 kubenswrapper[5619]:  >
Nov 01 06:11:06.591837 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:06.591616    5619 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 06:11:06.591837 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf_add83dd5-a128-499e-9e8b-00142cc7bc94_0(51534e5154139093b2f028913374ce5c67787f899d5a36728ae232589cd59703): error adding pod ai-tenant-e2e-extmod-b-3986bf_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"51534e5154139093b2f028913374ce5c67787f899d5a36728ae232589cd59703" Netns:"/var/run/netns/5f71eb12-93c5-4e14-9948-0604e0c5322f" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-extmod-b-3986bf;K8S_POD_NAME=payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq;K8S_POD_INFRA_CONTAINER_ID=51534e5154139093b2f028913374ce5c67787f899d5a36728ae232589cd59703;K8S_POD_UID=add83dd5-a128-499e-9e8b-00142cc7bc94" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq] networking: Multus: [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq/add83dd5-a128-499e-9e8b-00142cc7bc94]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-extmod-b-3986bf/pods/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq?timeout=1m0s": context deadline exceeded
Nov 01 06:11:06.591837 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:06.591837 ip-10-0-129-213 kubenswrapper[5619]:  > pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:06.591837 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:06.591640    5619 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 06:11:06.591837 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf_add83dd5-a128-499e-9e8b-00142cc7bc94_0(51534e5154139093b2f028913374ce5c67787f899d5a36728ae232589cd59703): error adding pod ai-tenant-e2e-extmod-b-3986bf_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"51534e5154139093b2f028913374ce5c67787f899d5a36728ae232589cd59703" Netns:"/var/run/netns/5f71eb12-93c5-4e14-9948-0604e0c5322f" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-extmod-b-3986bf;K8S_POD_NAME=payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq;K8S_POD_INFRA_CONTAINER_ID=51534e5154139093b2f028913374ce5c67787f899d5a36728ae232589cd59703;K8S_POD_UID=add83dd5-a128-499e-9e8b-00142cc7bc94" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq] networking: Multus: [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq/add83dd5-a128-499e-9e8b-00142cc7bc94]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-extmod-b-3986bf/pods/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq?timeout=1m0s": context deadline exceeded
Nov 01 06:11:06.591837 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:06.591837 ip-10-0-129-213 kubenswrapper[5619]:  > pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:06.591837 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:06.591732    5619 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf(add83dd5-a128-499e-9e8b-00142cc7bc94)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf(add83dd5-a128-499e-9e8b-00142cc7bc94)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf_add83dd5-a128-499e-9e8b-00142cc7bc94_0(51534e5154139093b2f028913374ce5c67787f899d5a36728ae232589cd59703): error adding pod ai-tenant-e2e-extmod-b-3986bf_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"51534e5154139093b2f028913374ce5c67787f899d5a36728ae232589cd59703\\\" Netns:\\\"/var/run/netns/5f71eb12-93c5-4e14-9948-0604e0c5322f\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-extmod-b-3986bf;K8S_POD_NAME=payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq;K8S_POD_INFRA_CONTAINER_ID=51534e5154139093b2f028913374ce5c67787f899d5a36728ae232589cd59703;K8S_POD_UID=add83dd5-a128-499e-9e8b-00142cc7bc94\\\" Path:\\\"\\\" ERRORED: error configuring pod [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq] networking: Multus: [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq/add83dd5-a128-499e-9e8b-00142cc7bc94]: error waiting for pod: Get \\\"https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-extmod-b-3986bf/pods/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq" podUID="add83dd5-a128-499e-9e8b-00142cc7bc94"
Nov 01 06:11:06.614610 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.614585    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 01 06:11:06.614724 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.614586    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 01 06:11:06.614724 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.614586    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 01 06:11:06.614724 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.614650    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qklrv\""
Nov 01 06:11:06.670048 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.670014    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"]
Nov 01 06:11:06.707394 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.707354    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.707561 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.707407    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3eb9ca80-f436-41b4-8354-ba7498f47580-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.707561 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.707427    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb9ca80-f436-41b4-8354-ba7498f47580-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.707636 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.707594    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.707680 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.707651    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.707784 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.707770    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.707855 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.707839    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdt6c\" (UniqueName: \"kubernetes.io/projected/3eb9ca80-f436-41b4-8354-ba7498f47580-kube-api-access-pdt6c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.809682 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.809585    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.809682 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.809650    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3eb9ca80-f436-41b4-8354-ba7498f47580-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.809682 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.809682    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb9ca80-f436-41b4-8354-ba7498f47580-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.809966 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.809746    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.809966 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.809795    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.809966 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.809886    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.809966 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.809950    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdt6c\" (UniqueName: \"kubernetes.io/projected/3eb9ca80-f436-41b4-8354-ba7498f47580-kube-api-access-pdt6c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.810670 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.810586    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.812295 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.812255    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb9ca80-f436-41b4-8354-ba7498f47580-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.812528 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.812484    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3eb9ca80-f436-41b4-8354-ba7498f47580-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.812528 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.812507    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.812654 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.812561    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.872140 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.872110    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.872426 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.872406    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdt6c\" (UniqueName: \"kubernetes.io/projected/3eb9ca80-f436-41b4-8354-ba7498f47580-kube-api-access-pdt6c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.875213 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.875185    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:06.927381 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.927348    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:06.927745 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:06.927728    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:08.848446 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:08.848405    5619 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 06:11:08.848446 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn_odh-ai-gateway-infra_d21d21dc-79f2-40c1-b701-e75e87bd144c_0(f0791435bf04456e01de19960b593a8aca6d01730d705ffe40a052b528a32517): error adding pod odh-ai-gateway-infra_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"f0791435bf04456e01de19960b593a8aca6d01730d705ffe40a052b528a32517" Netns:"/var/run/netns/4442c7b8-d416-4f4d-8081-6f91d228489f" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn;K8S_POD_INFRA_CONTAINER_ID=f0791435bf04456e01de19960b593a8aca6d01730d705ffe40a052b528a32517;K8S_POD_UID=d21d21dc-79f2-40c1-b701-e75e87bd144c" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn/d21d21dc-79f2-40c1-b701-e75e87bd144c]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn?timeout=1m0s": context deadline exceeded
Nov 01 06:11:08.848446 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:08.848446 ip-10-0-129-213 kubenswrapper[5619]:  >
Nov 01 06:11:08.848869 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:08.848464    5619 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 06:11:08.848869 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn_odh-ai-gateway-infra_d21d21dc-79f2-40c1-b701-e75e87bd144c_0(f0791435bf04456e01de19960b593a8aca6d01730d705ffe40a052b528a32517): error adding pod odh-ai-gateway-infra_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"f0791435bf04456e01de19960b593a8aca6d01730d705ffe40a052b528a32517" Netns:"/var/run/netns/4442c7b8-d416-4f4d-8081-6f91d228489f" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn;K8S_POD_INFRA_CONTAINER_ID=f0791435bf04456e01de19960b593a8aca6d01730d705ffe40a052b528a32517;K8S_POD_UID=d21d21dc-79f2-40c1-b701-e75e87bd144c" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn/d21d21dc-79f2-40c1-b701-e75e87bd144c]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn?timeout=1m0s": context deadline exceeded
Nov 01 06:11:08.848869 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:08.848869 ip-10-0-129-213 kubenswrapper[5619]:  > pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:08.848869 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:08.848484    5619 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 06:11:08.848869 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn_odh-ai-gateway-infra_d21d21dc-79f2-40c1-b701-e75e87bd144c_0(f0791435bf04456e01de19960b593a8aca6d01730d705ffe40a052b528a32517): error adding pod odh-ai-gateway-infra_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"f0791435bf04456e01de19960b593a8aca6d01730d705ffe40a052b528a32517" Netns:"/var/run/netns/4442c7b8-d416-4f4d-8081-6f91d228489f" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn;K8S_POD_INFRA_CONTAINER_ID=f0791435bf04456e01de19960b593a8aca6d01730d705ffe40a052b528a32517;K8S_POD_UID=d21d21dc-79f2-40c1-b701-e75e87bd144c" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn/d21d21dc-79f2-40c1-b701-e75e87bd144c]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn?timeout=1m0s": context deadline exceeded
Nov 01 06:11:08.848869 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:08.848869 ip-10-0-129-213 kubenswrapper[5619]:  > pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:08.848869 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:08.848585    5619 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn_odh-ai-gateway-infra(d21d21dc-79f2-40c1-b701-e75e87bd144c)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn_odh-ai-gateway-infra(d21d21dc-79f2-40c1-b701-e75e87bd144c)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn_odh-ai-gateway-infra_d21d21dc-79f2-40c1-b701-e75e87bd144c_0(f0791435bf04456e01de19960b593a8aca6d01730d705ffe40a052b528a32517): error adding pod odh-ai-gateway-infra_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"f0791435bf04456e01de19960b593a8aca6d01730d705ffe40a052b528a32517\\\" Netns:\\\"/var/run/netns/4442c7b8-d416-4f4d-8081-6f91d228489f\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn;K8S_POD_INFRA_CONTAINER_ID=f0791435bf04456e01de19960b593a8aca6d01730d705ffe40a052b528a32517;K8S_POD_UID=d21d21dc-79f2-40c1-b701-e75e87bd144c\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn/d21d21dc-79f2-40c1-b701-e75e87bd144c]: error waiting for pod: Get \\\"https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn" podUID="d21d21dc-79f2-40c1-b701-e75e87bd144c"
Nov 01 06:11:08.938895 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:08.938860    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:08.939344 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:08.939305    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:09.473867 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:09.473828    5619 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 06:11:09.473867 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_istiod-openshift-gateway-7ddd9cfdc8-c4ks9_openshift-ingress_3eb9ca80-f436-41b4-8354-ba7498f47580_0(5067287c797d8f87c938d23ab4e7eb3966bafb6472f211323b6d039fa2839036): error adding pod openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-c4ks9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"5067287c797d8f87c938d23ab4e7eb3966bafb6472f211323b6d039fa2839036" Netns:"/var/run/netns/95974e6b-5ad1-4de0-9b17-17f04ba6376a" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=istiod-openshift-gateway-7ddd9cfdc8-c4ks9;K8S_POD_INFRA_CONTAINER_ID=5067287c797d8f87c938d23ab4e7eb3966bafb6472f211323b6d039fa2839036;K8S_POD_UID=3eb9ca80-f436-41b4-8354-ba7498f47580" Path:"" ERRORED: error configuring pod [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9] networking: Multus: [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9/3eb9ca80-f436-41b4-8354-ba7498f47580]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-c4ks9?timeout=1m0s": context deadline exceeded
Nov 01 06:11:09.473867 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:09.473867 ip-10-0-129-213 kubenswrapper[5619]:  >
Nov 01 06:11:09.474166 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:09.473896    5619 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 06:11:09.474166 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_istiod-openshift-gateway-7ddd9cfdc8-c4ks9_openshift-ingress_3eb9ca80-f436-41b4-8354-ba7498f47580_0(5067287c797d8f87c938d23ab4e7eb3966bafb6472f211323b6d039fa2839036): error adding pod openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-c4ks9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"5067287c797d8f87c938d23ab4e7eb3966bafb6472f211323b6d039fa2839036" Netns:"/var/run/netns/95974e6b-5ad1-4de0-9b17-17f04ba6376a" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=istiod-openshift-gateway-7ddd9cfdc8-c4ks9;K8S_POD_INFRA_CONTAINER_ID=5067287c797d8f87c938d23ab4e7eb3966bafb6472f211323b6d039fa2839036;K8S_POD_UID=3eb9ca80-f436-41b4-8354-ba7498f47580" Path:"" ERRORED: error configuring pod [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9] networking: Multus: [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9/3eb9ca80-f436-41b4-8354-ba7498f47580]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-c4ks9?timeout=1m0s": context deadline exceeded
Nov 01 06:11:09.474166 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:09.474166 ip-10-0-129-213 kubenswrapper[5619]:  > pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:09.474166 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:09.473918    5619 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 06:11:09.474166 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_istiod-openshift-gateway-7ddd9cfdc8-c4ks9_openshift-ingress_3eb9ca80-f436-41b4-8354-ba7498f47580_0(5067287c797d8f87c938d23ab4e7eb3966bafb6472f211323b6d039fa2839036): error adding pod openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-c4ks9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"5067287c797d8f87c938d23ab4e7eb3966bafb6472f211323b6d039fa2839036" Netns:"/var/run/netns/95974e6b-5ad1-4de0-9b17-17f04ba6376a" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=istiod-openshift-gateway-7ddd9cfdc8-c4ks9;K8S_POD_INFRA_CONTAINER_ID=5067287c797d8f87c938d23ab4e7eb3966bafb6472f211323b6d039fa2839036;K8S_POD_UID=3eb9ca80-f436-41b4-8354-ba7498f47580" Path:"" ERRORED: error configuring pod [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9] networking: Multus: [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9/3eb9ca80-f436-41b4-8354-ba7498f47580]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-c4ks9?timeout=1m0s": context deadline exceeded
Nov 01 06:11:09.474166 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:09.474166 ip-10-0-129-213 kubenswrapper[5619]:  > pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:09.474166 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:09.474016    5619 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9_openshift-ingress(3eb9ca80-f436-41b4-8354-ba7498f47580)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"istiod-openshift-gateway-7ddd9cfdc8-c4ks9_openshift-ingress(3eb9ca80-f436-41b4-8354-ba7498f47580)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_istiod-openshift-gateway-7ddd9cfdc8-c4ks9_openshift-ingress_3eb9ca80-f436-41b4-8354-ba7498f47580_0(5067287c797d8f87c938d23ab4e7eb3966bafb6472f211323b6d039fa2839036): error adding pod openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-c4ks9 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"5067287c797d8f87c938d23ab4e7eb3966bafb6472f211323b6d039fa2839036\\\" Netns:\\\"/var/run/netns/95974e6b-5ad1-4de0-9b17-17f04ba6376a\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=istiod-openshift-gateway-7ddd9cfdc8-c4ks9;K8S_POD_INFRA_CONTAINER_ID=5067287c797d8f87c938d23ab4e7eb3966bafb6472f211323b6d039fa2839036;K8S_POD_UID=3eb9ca80-f436-41b4-8354-ba7498f47580\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9] networking: Multus: [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9/3eb9ca80-f436-41b4-8354-ba7498f47580]: error waiting for pod: Get \\\"https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-c4ks9?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9" podUID="3eb9ca80-f436-41b4-8354-ba7498f47580"
Nov 01 06:11:09.521534 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:09.521494    5619 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 06:11:09.521534 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf_add83dd5-a128-499e-9e8b-00142cc7bc94_0(6e36598cd1740fcefa394e9317ab365d5f60a93695f466a2aa7cce8aefd5c3d9): error adding pod ai-tenant-e2e-extmod-b-3986bf_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"6e36598cd1740fcefa394e9317ab365d5f60a93695f466a2aa7cce8aefd5c3d9" Netns:"/var/run/netns/855ef6ee-d46d-49e9-9089-c03b5407d92d" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-extmod-b-3986bf;K8S_POD_NAME=payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq;K8S_POD_INFRA_CONTAINER_ID=6e36598cd1740fcefa394e9317ab365d5f60a93695f466a2aa7cce8aefd5c3d9;K8S_POD_UID=add83dd5-a128-499e-9e8b-00142cc7bc94" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq] networking: Multus: [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq/add83dd5-a128-499e-9e8b-00142cc7bc94]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-extmod-b-3986bf/pods/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq?timeout=1m0s": context deadline exceeded
Nov 01 06:11:09.521534 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:09.521534 ip-10-0-129-213 kubenswrapper[5619]:  >
Nov 01 06:11:09.521707 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:09.521555    5619 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 06:11:09.521707 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf_add83dd5-a128-499e-9e8b-00142cc7bc94_0(6e36598cd1740fcefa394e9317ab365d5f60a93695f466a2aa7cce8aefd5c3d9): error adding pod ai-tenant-e2e-extmod-b-3986bf_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"6e36598cd1740fcefa394e9317ab365d5f60a93695f466a2aa7cce8aefd5c3d9" Netns:"/var/run/netns/855ef6ee-d46d-49e9-9089-c03b5407d92d" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-extmod-b-3986bf;K8S_POD_NAME=payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq;K8S_POD_INFRA_CONTAINER_ID=6e36598cd1740fcefa394e9317ab365d5f60a93695f466a2aa7cce8aefd5c3d9;K8S_POD_UID=add83dd5-a128-499e-9e8b-00142cc7bc94" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq] networking: Multus: [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq/add83dd5-a128-499e-9e8b-00142cc7bc94]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-extmod-b-3986bf/pods/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq?timeout=1m0s": context deadline exceeded
Nov 01 06:11:09.521707 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:09.521707 ip-10-0-129-213 kubenswrapper[5619]:  > pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:09.521707 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:09.521573    5619 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 06:11:09.521707 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf_add83dd5-a128-499e-9e8b-00142cc7bc94_0(6e36598cd1740fcefa394e9317ab365d5f60a93695f466a2aa7cce8aefd5c3d9): error adding pod ai-tenant-e2e-extmod-b-3986bf_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"6e36598cd1740fcefa394e9317ab365d5f60a93695f466a2aa7cce8aefd5c3d9" Netns:"/var/run/netns/855ef6ee-d46d-49e9-9089-c03b5407d92d" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-extmod-b-3986bf;K8S_POD_NAME=payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq;K8S_POD_INFRA_CONTAINER_ID=6e36598cd1740fcefa394e9317ab365d5f60a93695f466a2aa7cce8aefd5c3d9;K8S_POD_UID=add83dd5-a128-499e-9e8b-00142cc7bc94" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq] networking: Multus: [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq/add83dd5-a128-499e-9e8b-00142cc7bc94]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-extmod-b-3986bf/pods/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq?timeout=1m0s": context deadline exceeded
Nov 01 06:11:09.521707 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:09.521707 ip-10-0-129-213 kubenswrapper[5619]:  > pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:09.521707 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:09.521671    5619 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf(add83dd5-a128-499e-9e8b-00142cc7bc94)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf(add83dd5-a128-499e-9e8b-00142cc7bc94)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf_add83dd5-a128-499e-9e8b-00142cc7bc94_0(6e36598cd1740fcefa394e9317ab365d5f60a93695f466a2aa7cce8aefd5c3d9): error adding pod ai-tenant-e2e-extmod-b-3986bf_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"6e36598cd1740fcefa394e9317ab365d5f60a93695f466a2aa7cce8aefd5c3d9\\\" Netns:\\\"/var/run/netns/855ef6ee-d46d-49e9-9089-c03b5407d92d\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-extmod-b-3986bf;K8S_POD_NAME=payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq;K8S_POD_INFRA_CONTAINER_ID=6e36598cd1740fcefa394e9317ab365d5f60a93695f466a2aa7cce8aefd5c3d9;K8S_POD_UID=add83dd5-a128-499e-9e8b-00142cc7bc94\\\" Path:\\\"\\\" ERRORED: error configuring pod [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq] networking: Multus: [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq/add83dd5-a128-499e-9e8b-00142cc7bc94]: error waiting for pod: Get \\\"https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-extmod-b-3986bf/pods/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq" podUID="add83dd5-a128-499e-9e8b-00142cc7bc94"
Nov 01 06:11:09.839278 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:09.839159    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:09.946163 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:09.946132    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:09.946683 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:09.946608    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:10.067181 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:10.067147    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:10.311720 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:10.311678    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:10.451877 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:10.451827    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:11.543594 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:11.543558    5619 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 06:11:11.543594 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn_odh-ai-gateway-infra_d21d21dc-79f2-40c1-b701-e75e87bd144c_0(3ba1c7786bcd94a7c364aaadaabd228b329b50cec0d1670ce6cce7ea825c4e51): error adding pod odh-ai-gateway-infra_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3ba1c7786bcd94a7c364aaadaabd228b329b50cec0d1670ce6cce7ea825c4e51" Netns:"/var/run/netns/7f2cdb16-a03d-432a-9174-9d0cc58c6512" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn;K8S_POD_INFRA_CONTAINER_ID=3ba1c7786bcd94a7c364aaadaabd228b329b50cec0d1670ce6cce7ea825c4e51;K8S_POD_UID=d21d21dc-79f2-40c1-b701-e75e87bd144c" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn/d21d21dc-79f2-40c1-b701-e75e87bd144c]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn?timeout=1m0s": context deadline exceeded
Nov 01 06:11:11.543594 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:11.543594 ip-10-0-129-213 kubenswrapper[5619]:  >
Nov 01 06:11:11.543980 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:11.543617    5619 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 06:11:11.543980 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn_odh-ai-gateway-infra_d21d21dc-79f2-40c1-b701-e75e87bd144c_0(3ba1c7786bcd94a7c364aaadaabd228b329b50cec0d1670ce6cce7ea825c4e51): error adding pod odh-ai-gateway-infra_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3ba1c7786bcd94a7c364aaadaabd228b329b50cec0d1670ce6cce7ea825c4e51" Netns:"/var/run/netns/7f2cdb16-a03d-432a-9174-9d0cc58c6512" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn;K8S_POD_INFRA_CONTAINER_ID=3ba1c7786bcd94a7c364aaadaabd228b329b50cec0d1670ce6cce7ea825c4e51;K8S_POD_UID=d21d21dc-79f2-40c1-b701-e75e87bd144c" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn/d21d21dc-79f2-40c1-b701-e75e87bd144c]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn?timeout=1m0s": context deadline exceeded
Nov 01 06:11:11.543980 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:11.543980 ip-10-0-129-213 kubenswrapper[5619]:  > pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:11.543980 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:11.543635    5619 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 06:11:11.543980 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn_odh-ai-gateway-infra_d21d21dc-79f2-40c1-b701-e75e87bd144c_0(3ba1c7786bcd94a7c364aaadaabd228b329b50cec0d1670ce6cce7ea825c4e51): error adding pod odh-ai-gateway-infra_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3ba1c7786bcd94a7c364aaadaabd228b329b50cec0d1670ce6cce7ea825c4e51" Netns:"/var/run/netns/7f2cdb16-a03d-432a-9174-9d0cc58c6512" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn;K8S_POD_INFRA_CONTAINER_ID=3ba1c7786bcd94a7c364aaadaabd228b329b50cec0d1670ce6cce7ea825c4e51;K8S_POD_UID=d21d21dc-79f2-40c1-b701-e75e87bd144c" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn/d21d21dc-79f2-40c1-b701-e75e87bd144c]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn?timeout=1m0s": context deadline exceeded
Nov 01 06:11:11.543980 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:11.543980 ip-10-0-129-213 kubenswrapper[5619]:  > pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:11.543980 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:11.543721    5619 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn_odh-ai-gateway-infra(d21d21dc-79f2-40c1-b701-e75e87bd144c)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn_odh-ai-gateway-infra(d21d21dc-79f2-40c1-b701-e75e87bd144c)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn_odh-ai-gateway-infra_d21d21dc-79f2-40c1-b701-e75e87bd144c_0(3ba1c7786bcd94a7c364aaadaabd228b329b50cec0d1670ce6cce7ea825c4e51): error adding pod odh-ai-gateway-infra_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"3ba1c7786bcd94a7c364aaadaabd228b329b50cec0d1670ce6cce7ea825c4e51\\\" Netns:\\\"/var/run/netns/7f2cdb16-a03d-432a-9174-9d0cc58c6512\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn;K8S_POD_INFRA_CONTAINER_ID=3ba1c7786bcd94a7c364aaadaabd228b329b50cec0d1670ce6cce7ea825c4e51;K8S_POD_UID=d21d21dc-79f2-40c1-b701-e75e87bd144c\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn/d21d21dc-79f2-40c1-b701-e75e87bd144c]: error waiting for pod: Get \\\"https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn" podUID="d21d21dc-79f2-40c1-b701-e75e87bd144c"
Nov 01 06:11:11.629906 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:11.629859    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:11.770544 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:11.770508    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:12.036202 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:12.036167    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:12.543397 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:12.543361    5619 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 06:11:12.543397 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_istiod-openshift-gateway-7ddd9cfdc8-c4ks9_openshift-ingress_3eb9ca80-f436-41b4-8354-ba7498f47580_0(9e5efe57fba37f09c4247fecf0abdde0ec37bcd39c4db42e8ec6b9d096889ed9): error adding pod openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-c4ks9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"9e5efe57fba37f09c4247fecf0abdde0ec37bcd39c4db42e8ec6b9d096889ed9" Netns:"/var/run/netns/ee888db4-74a8-4db8-8e3b-a32a6c440972" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=istiod-openshift-gateway-7ddd9cfdc8-c4ks9;K8S_POD_INFRA_CONTAINER_ID=9e5efe57fba37f09c4247fecf0abdde0ec37bcd39c4db42e8ec6b9d096889ed9;K8S_POD_UID=3eb9ca80-f436-41b4-8354-ba7498f47580" Path:"" ERRORED: error configuring pod [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9] networking: Multus: [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9/3eb9ca80-f436-41b4-8354-ba7498f47580]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-c4ks9?timeout=1m0s": context deadline exceeded
Nov 01 06:11:12.543397 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:12.543397 ip-10-0-129-213 kubenswrapper[5619]:  >
Nov 01 06:11:12.543855 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:12.543416    5619 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 06:11:12.543855 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_istiod-openshift-gateway-7ddd9cfdc8-c4ks9_openshift-ingress_3eb9ca80-f436-41b4-8354-ba7498f47580_0(9e5efe57fba37f09c4247fecf0abdde0ec37bcd39c4db42e8ec6b9d096889ed9): error adding pod openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-c4ks9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"9e5efe57fba37f09c4247fecf0abdde0ec37bcd39c4db42e8ec6b9d096889ed9" Netns:"/var/run/netns/ee888db4-74a8-4db8-8e3b-a32a6c440972" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=istiod-openshift-gateway-7ddd9cfdc8-c4ks9;K8S_POD_INFRA_CONTAINER_ID=9e5efe57fba37f09c4247fecf0abdde0ec37bcd39c4db42e8ec6b9d096889ed9;K8S_POD_UID=3eb9ca80-f436-41b4-8354-ba7498f47580" Path:"" ERRORED: error configuring pod [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9] networking: Multus: [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9/3eb9ca80-f436-41b4-8354-ba7498f47580]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-c4ks9?timeout=1m0s": context deadline exceeded
Nov 01 06:11:12.543855 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:12.543855 ip-10-0-129-213 kubenswrapper[5619]:  > pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:12.543855 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:12.543440    5619 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 06:11:12.543855 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_istiod-openshift-gateway-7ddd9cfdc8-c4ks9_openshift-ingress_3eb9ca80-f436-41b4-8354-ba7498f47580_0(9e5efe57fba37f09c4247fecf0abdde0ec37bcd39c4db42e8ec6b9d096889ed9): error adding pod openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-c4ks9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"9e5efe57fba37f09c4247fecf0abdde0ec37bcd39c4db42e8ec6b9d096889ed9" Netns:"/var/run/netns/ee888db4-74a8-4db8-8e3b-a32a6c440972" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=istiod-openshift-gateway-7ddd9cfdc8-c4ks9;K8S_POD_INFRA_CONTAINER_ID=9e5efe57fba37f09c4247fecf0abdde0ec37bcd39c4db42e8ec6b9d096889ed9;K8S_POD_UID=3eb9ca80-f436-41b4-8354-ba7498f47580" Path:"" ERRORED: error configuring pod [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9] networking: Multus: [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9/3eb9ca80-f436-41b4-8354-ba7498f47580]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-c4ks9?timeout=1m0s": context deadline exceeded
Nov 01 06:11:12.543855 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:12.543855 ip-10-0-129-213 kubenswrapper[5619]:  > pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:12.543855 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:12.543527    5619 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9_openshift-ingress(3eb9ca80-f436-41b4-8354-ba7498f47580)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"istiod-openshift-gateway-7ddd9cfdc8-c4ks9_openshift-ingress(3eb9ca80-f436-41b4-8354-ba7498f47580)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_istiod-openshift-gateway-7ddd9cfdc8-c4ks9_openshift-ingress_3eb9ca80-f436-41b4-8354-ba7498f47580_0(9e5efe57fba37f09c4247fecf0abdde0ec37bcd39c4db42e8ec6b9d096889ed9): error adding pod openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-c4ks9 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"9e5efe57fba37f09c4247fecf0abdde0ec37bcd39c4db42e8ec6b9d096889ed9\\\" Netns:\\\"/var/run/netns/ee888db4-74a8-4db8-8e3b-a32a6c440972\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=istiod-openshift-gateway-7ddd9cfdc8-c4ks9;K8S_POD_INFRA_CONTAINER_ID=9e5efe57fba37f09c4247fecf0abdde0ec37bcd39c4db42e8ec6b9d096889ed9;K8S_POD_UID=3eb9ca80-f436-41b4-8354-ba7498f47580\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9] networking: Multus: [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9/3eb9ca80-f436-41b4-8354-ba7498f47580]: error waiting for pod: Get \\\"https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-c4ks9?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9" podUID="3eb9ca80-f436-41b4-8354-ba7498f47580"
Nov 01 06:11:16.503315 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:16.503280    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:16.934821 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:16.934784    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:19.925485 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:19.925436    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:20.228782 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:20.228671    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:20.374347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:20.374269    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:21.954645 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:21.954604    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:21.955118 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:21.955073    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:23.451389 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:23.451353    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:23.483461 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:23.483428    5619 patch_prober.go:29] interesting pod/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:11:23.483642 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:23.483506    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv" podUID="39ae3e00-d79f-473e-ae51-c5b5029f5353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:11:23.954617 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:23.954585    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:23.954961 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:23.954944    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:24.560002 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:24.559964    5619 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 06:11:24.560002 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf_add83dd5-a128-499e-9e8b-00142cc7bc94_0(f0db8438f0d386c470509f62e11fc61acc768eab6929e8a44d206844dee867a4): error adding pod ai-tenant-e2e-extmod-b-3986bf_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"f0db8438f0d386c470509f62e11fc61acc768eab6929e8a44d206844dee867a4" Netns:"/var/run/netns/a9eefe1b-cd03-4635-82cc-7d6aac134807" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-extmod-b-3986bf;K8S_POD_NAME=payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq;K8S_POD_INFRA_CONTAINER_ID=f0db8438f0d386c470509f62e11fc61acc768eab6929e8a44d206844dee867a4;K8S_POD_UID=add83dd5-a128-499e-9e8b-00142cc7bc94" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq] networking: Multus: [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq/add83dd5-a128-499e-9e8b-00142cc7bc94]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-extmod-b-3986bf/pods/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq?timeout=1m0s": context deadline exceeded
Nov 01 06:11:24.560002 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:24.560002 ip-10-0-129-213 kubenswrapper[5619]:  >
Nov 01 06:11:24.560416 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:24.560031    5619 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 06:11:24.560416 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf_add83dd5-a128-499e-9e8b-00142cc7bc94_0(f0db8438f0d386c470509f62e11fc61acc768eab6929e8a44d206844dee867a4): error adding pod ai-tenant-e2e-extmod-b-3986bf_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"f0db8438f0d386c470509f62e11fc61acc768eab6929e8a44d206844dee867a4" Netns:"/var/run/netns/a9eefe1b-cd03-4635-82cc-7d6aac134807" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-extmod-b-3986bf;K8S_POD_NAME=payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq;K8S_POD_INFRA_CONTAINER_ID=f0db8438f0d386c470509f62e11fc61acc768eab6929e8a44d206844dee867a4;K8S_POD_UID=add83dd5-a128-499e-9e8b-00142cc7bc94" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq] networking: Multus: [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq/add83dd5-a128-499e-9e8b-00142cc7bc94]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-extmod-b-3986bf/pods/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq?timeout=1m0s": context deadline exceeded
Nov 01 06:11:24.560416 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:24.560416 ip-10-0-129-213 kubenswrapper[5619]:  > pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:24.560416 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:24.560059    5619 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 06:11:24.560416 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf_add83dd5-a128-499e-9e8b-00142cc7bc94_0(f0db8438f0d386c470509f62e11fc61acc768eab6929e8a44d206844dee867a4): error adding pod ai-tenant-e2e-extmod-b-3986bf_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"f0db8438f0d386c470509f62e11fc61acc768eab6929e8a44d206844dee867a4" Netns:"/var/run/netns/a9eefe1b-cd03-4635-82cc-7d6aac134807" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-extmod-b-3986bf;K8S_POD_NAME=payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq;K8S_POD_INFRA_CONTAINER_ID=f0db8438f0d386c470509f62e11fc61acc768eab6929e8a44d206844dee867a4;K8S_POD_UID=add83dd5-a128-499e-9e8b-00142cc7bc94" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq] networking: Multus: [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq/add83dd5-a128-499e-9e8b-00142cc7bc94]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-extmod-b-3986bf/pods/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq?timeout=1m0s": context deadline exceeded
Nov 01 06:11:24.560416 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:24.560416 ip-10-0-129-213 kubenswrapper[5619]:  > pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:24.560416 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:24.560169    5619 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf(add83dd5-a128-499e-9e8b-00142cc7bc94)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf(add83dd5-a128-499e-9e8b-00142cc7bc94)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf_add83dd5-a128-499e-9e8b-00142cc7bc94_0(f0db8438f0d386c470509f62e11fc61acc768eab6929e8a44d206844dee867a4): error adding pod ai-tenant-e2e-extmod-b-3986bf_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"f0db8438f0d386c470509f62e11fc61acc768eab6929e8a44d206844dee867a4\\\" Netns:\\\"/var/run/netns/a9eefe1b-cd03-4635-82cc-7d6aac134807\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-extmod-b-3986bf;K8S_POD_NAME=payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq;K8S_POD_INFRA_CONTAINER_ID=f0db8438f0d386c470509f62e11fc61acc768eab6929e8a44d206844dee867a4;K8S_POD_UID=add83dd5-a128-499e-9e8b-00142cc7bc94\\\" Path:\\\"\\\" ERRORED: error configuring pod [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq] networking: Multus: [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq/add83dd5-a128-499e-9e8b-00142cc7bc94]: error waiting for pod: Get \\\"https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-extmod-b-3986bf/pods/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq" podUID="add83dd5-a128-499e-9e8b-00142cc7bc94"
Nov 01 06:11:25.046613 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:25.046580    5619 generic.go:356] "Generic (PLEG): container finished" podID="39ae3e00-d79f-473e-ae51-c5b5029f5353" containerID="5e37f4871a4556972d0d93b79f58144305d7bbe0a03343b70e755935605b7b34" exitCode=0
Nov 01 06:11:25.046788 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:25.046607    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv" event={"ID":"39ae3e00-d79f-473e-ae51-c5b5029f5353","Type":"ContainerDied","Data":"5e37f4871a4556972d0d93b79f58144305d7bbe0a03343b70e755935605b7b34"}
Nov 01 06:11:25.047091 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:25.047072    5619 scope.go:122] "RemoveContainer" containerID="5e37f4871a4556972d0d93b79f58144305d7bbe0a03343b70e755935605b7b34"
Nov 01 06:11:25.048759 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:25.048713    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:11:25.048879 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:25.048820    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:11:25.048879 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:25.048873    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:11:25.899841 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:25.899804    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:11:26.080348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:26.077548    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv" event={"ID":"39ae3e00-d79f-473e-ae51-c5b5029f5353","Type":"ContainerStarted","Data":"5941754270fab4969e3f000da9d354c983bf67936511f09cc2920a305361fe12"}
Nov 01 06:11:26.586311 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:26.586272    5619 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 06:11:26.586311 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn_odh-ai-gateway-infra_d21d21dc-79f2-40c1-b701-e75e87bd144c_0(f74fcadd32d150191b7d47534c072fdf22d4ad7529cf94c7e29ca7ca5c93b412): error adding pod odh-ai-gateway-infra_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"f74fcadd32d150191b7d47534c072fdf22d4ad7529cf94c7e29ca7ca5c93b412" Netns:"/var/run/netns/956168c0-130e-4ed7-a3ad-c415e2ad8672" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn;K8S_POD_INFRA_CONTAINER_ID=f74fcadd32d150191b7d47534c072fdf22d4ad7529cf94c7e29ca7ca5c93b412;K8S_POD_UID=d21d21dc-79f2-40c1-b701-e75e87bd144c" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn/d21d21dc-79f2-40c1-b701-e75e87bd144c]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn?timeout=1m0s": context deadline exceeded
Nov 01 06:11:26.586311 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:26.586311 ip-10-0-129-213 kubenswrapper[5619]:  >
Nov 01 06:11:26.586639 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:26.586362    5619 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 06:11:26.586639 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn_odh-ai-gateway-infra_d21d21dc-79f2-40c1-b701-e75e87bd144c_0(f74fcadd32d150191b7d47534c072fdf22d4ad7529cf94c7e29ca7ca5c93b412): error adding pod odh-ai-gateway-infra_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"f74fcadd32d150191b7d47534c072fdf22d4ad7529cf94c7e29ca7ca5c93b412" Netns:"/var/run/netns/956168c0-130e-4ed7-a3ad-c415e2ad8672" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn;K8S_POD_INFRA_CONTAINER_ID=f74fcadd32d150191b7d47534c072fdf22d4ad7529cf94c7e29ca7ca5c93b412;K8S_POD_UID=d21d21dc-79f2-40c1-b701-e75e87bd144c" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn/d21d21dc-79f2-40c1-b701-e75e87bd144c]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn?timeout=1m0s": context deadline exceeded
Nov 01 06:11:26.586639 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:26.586639 ip-10-0-129-213 kubenswrapper[5619]:  > pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:26.586639 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:26.586383    5619 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 06:11:26.586639 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn_odh-ai-gateway-infra_d21d21dc-79f2-40c1-b701-e75e87bd144c_0(f74fcadd32d150191b7d47534c072fdf22d4ad7529cf94c7e29ca7ca5c93b412): error adding pod odh-ai-gateway-infra_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"f74fcadd32d150191b7d47534c072fdf22d4ad7529cf94c7e29ca7ca5c93b412" Netns:"/var/run/netns/956168c0-130e-4ed7-a3ad-c415e2ad8672" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn;K8S_POD_INFRA_CONTAINER_ID=f74fcadd32d150191b7d47534c072fdf22d4ad7529cf94c7e29ca7ca5c93b412;K8S_POD_UID=d21d21dc-79f2-40c1-b701-e75e87bd144c" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn/d21d21dc-79f2-40c1-b701-e75e87bd144c]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn?timeout=1m0s": context deadline exceeded
Nov 01 06:11:26.586639 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:26.586639 ip-10-0-129-213 kubenswrapper[5619]:  > pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:26.586639 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:26.586455    5619 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn_odh-ai-gateway-infra(d21d21dc-79f2-40c1-b701-e75e87bd144c)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn_odh-ai-gateway-infra(d21d21dc-79f2-40c1-b701-e75e87bd144c)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn_odh-ai-gateway-infra_d21d21dc-79f2-40c1-b701-e75e87bd144c_0(f74fcadd32d150191b7d47534c072fdf22d4ad7529cf94c7e29ca7ca5c93b412): error adding pod odh-ai-gateway-infra_maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"f74fcadd32d150191b7d47534c072fdf22d4ad7529cf94c7e29ca7ca5c93b412\\\" Netns:\\\"/var/run/netns/956168c0-130e-4ed7-a3ad-c415e2ad8672\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn;K8S_POD_INFRA_CONTAINER_ID=f74fcadd32d150191b7d47534c072fdf22d4ad7529cf94c7e29ca7ca5c93b412;K8S_POD_UID=d21d21dc-79f2-40c1-b701-e75e87bd144c\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn] networking: Multus: [odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn/d21d21dc-79f2-40c1-b701-e75e87bd144c]: error waiting for pod: Get \\\"https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn" podUID="d21d21dc-79f2-40c1-b701-e75e87bd144c"
Nov 01 06:11:26.632936 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:26.632905    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:11:26.633184 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:26.633158    5619 patch_prober.go:29] interesting pod/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 06:11:26.633241 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:26.633204    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv" podUID="39ae3e00-d79f-473e-ae51-c5b5029f5353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 06:11:26.961626 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:26.961442    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:26.967918 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:26.967671    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:27.633838 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:27.633803    5619 patch_prober.go:29] interesting pod/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 06:11:27.634053 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:27.633855    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv" podUID="39ae3e00-d79f-473e-ae51-c5b5029f5353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 06:11:28.285684 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:28.285650    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:28.633676 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:28.633639    5619 patch_prober.go:29] interesting pod/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 06:11:28.633861 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:28.633697    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv" podUID="39ae3e00-d79f-473e-ae51-c5b5029f5353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 06:11:29.595236 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:29.595198    5619 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 06:11:29.595236 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_istiod-openshift-gateway-7ddd9cfdc8-c4ks9_openshift-ingress_3eb9ca80-f436-41b4-8354-ba7498f47580_0(1d718676042d2ffed537a3349cad3a05f4392f1df8bac56857f4011be4b46260): error adding pod openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-c4ks9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"1d718676042d2ffed537a3349cad3a05f4392f1df8bac56857f4011be4b46260" Netns:"/var/run/netns/16a48827-a869-4569-9ac4-b9ee360bd904" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=istiod-openshift-gateway-7ddd9cfdc8-c4ks9;K8S_POD_INFRA_CONTAINER_ID=1d718676042d2ffed537a3349cad3a05f4392f1df8bac56857f4011be4b46260;K8S_POD_UID=3eb9ca80-f436-41b4-8354-ba7498f47580" Path:"" ERRORED: error configuring pod [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9] networking: Multus: [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9/3eb9ca80-f436-41b4-8354-ba7498f47580]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-c4ks9?timeout=1m0s": context deadline exceeded
Nov 01 06:11:29.595236 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:29.595236 ip-10-0-129-213 kubenswrapper[5619]:  >
Nov 01 06:11:29.595672 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:29.595262    5619 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 06:11:29.595672 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_istiod-openshift-gateway-7ddd9cfdc8-c4ks9_openshift-ingress_3eb9ca80-f436-41b4-8354-ba7498f47580_0(1d718676042d2ffed537a3349cad3a05f4392f1df8bac56857f4011be4b46260): error adding pod openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-c4ks9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"1d718676042d2ffed537a3349cad3a05f4392f1df8bac56857f4011be4b46260" Netns:"/var/run/netns/16a48827-a869-4569-9ac4-b9ee360bd904" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=istiod-openshift-gateway-7ddd9cfdc8-c4ks9;K8S_POD_INFRA_CONTAINER_ID=1d718676042d2ffed537a3349cad3a05f4392f1df8bac56857f4011be4b46260;K8S_POD_UID=3eb9ca80-f436-41b4-8354-ba7498f47580" Path:"" ERRORED: error configuring pod [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9] networking: Multus: [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9/3eb9ca80-f436-41b4-8354-ba7498f47580]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-c4ks9?timeout=1m0s": context deadline exceeded
Nov 01 06:11:29.595672 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:29.595672 ip-10-0-129-213 kubenswrapper[5619]:  > pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:29.595672 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:29.595285    5619 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 06:11:29.595672 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_istiod-openshift-gateway-7ddd9cfdc8-c4ks9_openshift-ingress_3eb9ca80-f436-41b4-8354-ba7498f47580_0(1d718676042d2ffed537a3349cad3a05f4392f1df8bac56857f4011be4b46260): error adding pod openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-c4ks9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"1d718676042d2ffed537a3349cad3a05f4392f1df8bac56857f4011be4b46260" Netns:"/var/run/netns/16a48827-a869-4569-9ac4-b9ee360bd904" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=istiod-openshift-gateway-7ddd9cfdc8-c4ks9;K8S_POD_INFRA_CONTAINER_ID=1d718676042d2ffed537a3349cad3a05f4392f1df8bac56857f4011be4b46260;K8S_POD_UID=3eb9ca80-f436-41b4-8354-ba7498f47580" Path:"" ERRORED: error configuring pod [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9] networking: Multus: [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9/3eb9ca80-f436-41b4-8354-ba7498f47580]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-c4ks9?timeout=1m0s": context deadline exceeded
Nov 01 06:11:29.595672 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:29.595672 ip-10-0-129-213 kubenswrapper[5619]:  > pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:29.595672 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:29.595402    5619 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"istiod-openshift-gateway-7ddd9cfdc8-c4ks9_openshift-ingress(3eb9ca80-f436-41b4-8354-ba7498f47580)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"istiod-openshift-gateway-7ddd9cfdc8-c4ks9_openshift-ingress(3eb9ca80-f436-41b4-8354-ba7498f47580)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_istiod-openshift-gateway-7ddd9cfdc8-c4ks9_openshift-ingress_3eb9ca80-f436-41b4-8354-ba7498f47580_0(1d718676042d2ffed537a3349cad3a05f4392f1df8bac56857f4011be4b46260): error adding pod openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-c4ks9 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"1d718676042d2ffed537a3349cad3a05f4392f1df8bac56857f4011be4b46260\\\" Netns:\\\"/var/run/netns/16a48827-a869-4569-9ac4-b9ee360bd904\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=istiod-openshift-gateway-7ddd9cfdc8-c4ks9;K8S_POD_INFRA_CONTAINER_ID=1d718676042d2ffed537a3349cad3a05f4392f1df8bac56857f4011be4b46260;K8S_POD_UID=3eb9ca80-f436-41b4-8354-ba7498f47580\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9] networking: Multus: [openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9/3eb9ca80-f436-41b4-8354-ba7498f47580]: error waiting for pod: Get \\\"https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/istiod-openshift-gateway-7ddd9cfdc8-c4ks9?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9" podUID="3eb9ca80-f436-41b4-8354-ba7498f47580"
Nov 01 06:11:29.633380 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:29.633350    5619 patch_prober.go:29] interesting pod/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 06:11:29.633536 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:29.633407    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv" podUID="39ae3e00-d79f-473e-ae51-c5b5029f5353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 06:11:30.633162 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:30.633118    5619 patch_prober.go:29] interesting pod/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 01 06:11:30.633695 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:30.633177    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv" podUID="39ae3e00-d79f-473e-ae51-c5b5029f5353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 01 06:11:31.637486 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:31.637450    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:11:31.637951 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:31.637931    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:11:31.638779 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:31.638754    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:11:33.160443 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:33.160402    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:35.954490 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:35.954450    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:35.954981 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:35.954959    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:37.814494 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:37.814462    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:37.955096 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:37.955057    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:37.955575 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:37.955556    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:38.004605 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:38.004573    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:38.214682 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:38.214606    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:38.551878 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:38.551838    5619 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 01 06:11:38.551878 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf_add83dd5-a128-499e-9e8b-00142cc7bc94_0(d28a55615fbf61134b93bbab425491e73f21f2b1e15fab7b002c1ea5e65403ea): error adding pod ai-tenant-e2e-extmod-b-3986bf_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"d28a55615fbf61134b93bbab425491e73f21f2b1e15fab7b002c1ea5e65403ea" Netns:"/var/run/netns/a5068e6f-5eb0-4b57-b6ad-ec9c19aa9623" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-extmod-b-3986bf;K8S_POD_NAME=payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq;K8S_POD_INFRA_CONTAINER_ID=d28a55615fbf61134b93bbab425491e73f21f2b1e15fab7b002c1ea5e65403ea;K8S_POD_UID=add83dd5-a128-499e-9e8b-00142cc7bc94" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq] networking: Multus: [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq/add83dd5-a128-499e-9e8b-00142cc7bc94]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-extmod-b-3986bf/pods/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq?timeout=1m0s": context deadline exceeded
Nov 01 06:11:38.551878 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:38.551878 ip-10-0-129-213 kubenswrapper[5619]:  >
Nov 01 06:11:38.552152 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:38.551898    5619 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 01 06:11:38.552152 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf_add83dd5-a128-499e-9e8b-00142cc7bc94_0(d28a55615fbf61134b93bbab425491e73f21f2b1e15fab7b002c1ea5e65403ea): error adding pod ai-tenant-e2e-extmod-b-3986bf_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"d28a55615fbf61134b93bbab425491e73f21f2b1e15fab7b002c1ea5e65403ea" Netns:"/var/run/netns/a5068e6f-5eb0-4b57-b6ad-ec9c19aa9623" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-extmod-b-3986bf;K8S_POD_NAME=payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq;K8S_POD_INFRA_CONTAINER_ID=d28a55615fbf61134b93bbab425491e73f21f2b1e15fab7b002c1ea5e65403ea;K8S_POD_UID=add83dd5-a128-499e-9e8b-00142cc7bc94" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq] networking: Multus: [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq/add83dd5-a128-499e-9e8b-00142cc7bc94]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-extmod-b-3986bf/pods/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq?timeout=1m0s": context deadline exceeded
Nov 01 06:11:38.552152 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:38.552152 ip-10-0-129-213 kubenswrapper[5619]:  > pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:38.552152 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:38.551921    5619 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 01 06:11:38.552152 ip-10-0-129-213 kubenswrapper[5619]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf_add83dd5-a128-499e-9e8b-00142cc7bc94_0(d28a55615fbf61134b93bbab425491e73f21f2b1e15fab7b002c1ea5e65403ea): error adding pod ai-tenant-e2e-extmod-b-3986bf_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"d28a55615fbf61134b93bbab425491e73f21f2b1e15fab7b002c1ea5e65403ea" Netns:"/var/run/netns/a5068e6f-5eb0-4b57-b6ad-ec9c19aa9623" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-extmod-b-3986bf;K8S_POD_NAME=payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq;K8S_POD_INFRA_CONTAINER_ID=d28a55615fbf61134b93bbab425491e73f21f2b1e15fab7b002c1ea5e65403ea;K8S_POD_UID=add83dd5-a128-499e-9e8b-00142cc7bc94" Path:"" ERRORED: error configuring pod [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq] networking: Multus: [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq/add83dd5-a128-499e-9e8b-00142cc7bc94]: error waiting for pod: Get "https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-extmod-b-3986bf/pods/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq?timeout=1m0s": context deadline exceeded
Nov 01 06:11:38.552152 ip-10-0-129-213 kubenswrapper[5619]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 01 06:11:38.552152 ip-10-0-129-213 kubenswrapper[5619]:  > pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:38.552152 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:38.551998    5619 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf(add83dd5-a128-499e-9e8b-00142cc7bc94)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf(add83dd5-a128-499e-9e8b-00142cc7bc94)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq_ai-tenant-e2e-extmod-b-3986bf_add83dd5-a128-499e-9e8b-00142cc7bc94_0(d28a55615fbf61134b93bbab425491e73f21f2b1e15fab7b002c1ea5e65403ea): error adding pod ai-tenant-e2e-extmod-b-3986bf_payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"d28a55615fbf61134b93bbab425491e73f21f2b1e15fab7b002c1ea5e65403ea\\\" Netns:\\\"/var/run/netns/a5068e6f-5eb0-4b57-b6ad-ec9c19aa9623\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=ai-tenant-e2e-extmod-b-3986bf;K8S_POD_NAME=payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq;K8S_POD_INFRA_CONTAINER_ID=d28a55615fbf61134b93bbab425491e73f21f2b1e15fab7b002c1ea5e65403ea;K8S_POD_UID=add83dd5-a128-499e-9e8b-00142cc7bc94\\\" Path:\\\"\\\" ERRORED: error configuring pod [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq] networking: Multus: [ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq/add83dd5-a128-499e-9e8b-00142cc7bc94]: error waiting for pod: Get \\\"https://a902e330db8824c50ae5cc9eea4facf0-230a45cd30a76d02.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/ai-tenant-e2e-extmod-b-3986bf/pods/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq" podUID="add83dd5-a128-499e-9e8b-00142cc7bc94"
Nov 01 06:11:39.636815 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:39.636777    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"]
Nov 01 06:11:39.637208 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:11:39.636920    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21d21dc_79f2_40c1_b701_e75e87bd144c.slice/crio-d626e477f8209c3302f2ff41054fe874b6ac8ae2d23c1e1605b73da5c6d35511 WatchSource:0}: Error finding container d626e477f8209c3302f2ff41054fe874b6ac8ae2d23c1e1605b73da5c6d35511: Status 404 returned error can't find the container with id d626e477f8209c3302f2ff41054fe874b6ac8ae2d23c1e1605b73da5c6d35511
Nov 01 06:11:40.172363 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:40.172308    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn" event={"ID":"d21d21dc-79f2-40c1-b701-e75e87bd144c","Type":"ContainerStarted","Data":"d626e477f8209c3302f2ff41054fe874b6ac8ae2d23c1e1605b73da5c6d35511"}
Nov 01 06:11:41.179549 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:41.179511    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn" event={"ID":"d21d21dc-79f2-40c1-b701-e75e87bd144c","Type":"ContainerStarted","Data":"cfd4f9b85f953e5b599a89f45c846b7fbe798187c2fe583967d66fbc4de3b609"}
Nov 01 06:11:41.180031 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:41.179604    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:41.298429 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:41.298373    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn" podStartSLOduration=35.491034823 podStartE2EDuration="36.298360385s" totalImagesPullingTime="807.325562ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:11:41.297819291 +0000 UTC m=+1538.508642802" watchObservedRunningTime="2026-11-01 06:11:41.298360385 +0000 UTC m=+1538.509183935"
Nov 01 06:11:42.856233 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:42.856195    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:42.960435 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:42.960400    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:42.960847 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:42.960829    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:43.393364 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:43.393306    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"]
Nov 01 06:11:43.433168 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:11:43.433135    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb9ca80_f436_41b4_8354_ba7498f47580.slice/crio-6fce4672e2f4193d7f23938c357ce3dfd655ed7a9f16637904268194adf64d50 WatchSource:0}: Error finding container 6fce4672e2f4193d7f23938c357ce3dfd655ed7a9f16637904268194adf64d50: Status 404 returned error can't find the container with id 6fce4672e2f4193d7f23938c357ce3dfd655ed7a9f16637904268194adf64d50
Nov 01 06:11:44.199581 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:44.199544    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9" event={"ID":"3eb9ca80-f436-41b4-8354-ba7498f47580","Type":"ContainerStarted","Data":"6fce4672e2f4193d7f23938c357ce3dfd655ed7a9f16637904268194adf64d50"}
Nov 01 06:11:46.985744 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:46.985710    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:11:46.986046 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:46.985776    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:11:47.194071 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:47.194039    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:11:47.225018 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:47.224986    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9" event={"ID":"3eb9ca80-f436-41b4-8354-ba7498f47580","Type":"ContainerStarted","Data":"01cad546c840c7fd3f0f85d7d8739d4e5501c2aa8041bf7e570079458feede78"}
Nov 01 06:11:47.225205 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:47.225105    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:47.440275 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:47.440226    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9" podStartSLOduration=37.891265484 podStartE2EDuration="41.440209332s" totalImagesPullingTime="3.548943848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:11:47.438377871 +0000 UTC m=+1544.649201380" watchObservedRunningTime="2026-11-01 06:11:47.440209332 +0000 UTC m=+1544.651032841"
Nov 01 06:11:47.671045 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:47.671009    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:48.236203 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:48.236065    5619 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-c4ks9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:11:48.236203 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:48.236123    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9" podUID="3eb9ca80-f436-41b4-8354-ba7498f47580" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:11:49.139882 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:49.139839    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:49.283866 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:49.283830    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:49.686153 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:49.686105    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:49.825910 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:49.825870    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:51.233355 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:51.233309    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:11:52.470259 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:52.470220    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:52.959166 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:52.959134    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:52.959518 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:52.959502    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:53.338045 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:53.338015    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"]
Nov 01 06:11:53.338735 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:11:53.338705    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd83dd5_a128_499e_9e8b_00142cc7bc94.slice/crio-e3c035b2c313506e728379d8502473821b6bc4604c481755707241507abbf9e9 WatchSource:0}: Error finding container e3c035b2c313506e728379d8502473821b6bc4604c481755707241507abbf9e9: Status 404 returned error can't find the container with id e3c035b2c313506e728379d8502473821b6bc4604c481755707241507abbf9e9
Nov 01 06:11:54.271039 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:54.271002    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq" event={"ID":"add83dd5-a128-499e-9e8b-00142cc7bc94","Type":"ContainerStarted","Data":"6bb73475d0f0e8a130b118beaab7918ff1008f83e33e8325db30469bfaf657db"}
Nov 01 06:11:54.271574 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:54.271090    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq" event={"ID":"add83dd5-a128-499e-9e8b-00142cc7bc94","Type":"ContainerStarted","Data":"e3c035b2c313506e728379d8502473821b6bc4604c481755707241507abbf9e9"}
Nov 01 06:11:54.271574 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:54.271219    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:11:54.391549 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:54.391494    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq" podStartSLOduration=51.391480981 podStartE2EDuration="51.391480981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:11:54.389195469 +0000 UTC m=+1551.600018978" watchObservedRunningTime="2026-11-01 06:11:54.391480981 +0000 UTC m=+1551.602304489"
Nov 01 06:11:54.742815 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:54.742775    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:11:55.880138 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:11:55.880094    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:11:57.397317 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:11:57.397267    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:00.616050 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:00.616013    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:02.280780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:02.280742    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:05.280632 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:05.280602    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:12:05.453078 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:05.453042    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"]
Nov 01 06:12:07.512079 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:07.512038    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:10.453401 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:10.453360    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7" podUID="e59595a9-a730-43f0-a4cd-c667b99f57d6" containerName="payload-processing" containerID="cri-o://93c2f3fe9d5c55bb8793e0622633a7d82aa5e47e03aca034c43b28bed7287eef" gracePeriod=25
Nov 01 06:12:10.722655 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:10.722578    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"
Nov 01 06:12:10.739852 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:10.739815    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:10.793878 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:10.793830    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e59595a9-a730-43f0-a4cd-c667b99f57d6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e59595a9-a730-43f0-a4cd-c667b99f57d6-routing-overlay-volume\") pod \"e59595a9-a730-43f0-a4cd-c667b99f57d6\" (UID: \"e59595a9-a730-43f0-a4cd-c667b99f57d6\") "
Nov 01 06:12:10.794167 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:10.793924    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e59595a9-a730-43f0-a4cd-c667b99f57d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59595a9-a730-43f0-a4cd-c667b99f57d6-plugins-config-volume\") pod \"e59595a9-a730-43f0-a4cd-c667b99f57d6\" (UID: \"e59595a9-a730-43f0-a4cd-c667b99f57d6\") "
Nov 01 06:12:10.794167 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:10.794057    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e59595a9-a730-43f0-a4cd-c667b99f57d6-kube-api-access-x46bv\" (UniqueName: \"kubernetes.io/projected/e59595a9-a730-43f0-a4cd-c667b99f57d6-kube-api-access-x46bv\") pod \"e59595a9-a730-43f0-a4cd-c667b99f57d6\" (UID: \"e59595a9-a730-43f0-a4cd-c667b99f57d6\") "
Nov 01 06:12:10.794167 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:10.794114    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e59595a9-a730-43f0-a4cd-c667b99f57d6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e59595a9-a730-43f0-a4cd-c667b99f57d6-provider-credentials\") pod \"e59595a9-a730-43f0-a4cd-c667b99f57d6\" (UID: \"e59595a9-a730-43f0-a4cd-c667b99f57d6\") "
Nov 01 06:12:10.794897 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:10.794852    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59595a9-a730-43f0-a4cd-c667b99f57d6-plugins-config-volume" pod "e59595a9-a730-43f0-a4cd-c667b99f57d6" (UID: "e59595a9-a730-43f0-a4cd-c667b99f57d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:12:10.795190 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:10.795161    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59595a9-a730-43f0-a4cd-c667b99f57d6-routing-overlay-volume" pod "e59595a9-a730-43f0-a4cd-c667b99f57d6" (UID: "e59595a9-a730-43f0-a4cd-c667b99f57d6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:12:10.797583 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:10.797551    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59595a9-a730-43f0-a4cd-c667b99f57d6-provider-credentials" pod "e59595a9-a730-43f0-a4cd-c667b99f57d6" (UID: "e59595a9-a730-43f0-a4cd-c667b99f57d6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:12:10.803559 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:10.803526    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59595a9-a730-43f0-a4cd-c667b99f57d6-kube-api-access-x46bv" pod "e59595a9-a730-43f0-a4cd-c667b99f57d6" (UID: "e59595a9-a730-43f0-a4cd-c667b99f57d6"). InnerVolumeSpecName "kube-api-access-x46bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:12:10.896560 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:10.896529    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e59595a9-a730-43f0-a4cd-c667b99f57d6-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:10.896560 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:10.896558    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59595a9-a730-43f0-a4cd-c667b99f57d6-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:10.896757 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:10.896568    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x46bv\" (UniqueName: \"kubernetes.io/projected/e59595a9-a730-43f0-a4cd-c667b99f57d6-kube-api-access-x46bv\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:10.896757 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:10.896577    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e59595a9-a730-43f0-a4cd-c667b99f57d6-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:11.381069 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:11.381035    5619 generic.go:356] "Generic (PLEG): container finished" podID="e59595a9-a730-43f0-a4cd-c667b99f57d6" containerID="93c2f3fe9d5c55bb8793e0622633a7d82aa5e47e03aca034c43b28bed7287eef" exitCode=0
Nov 01 06:12:11.381220 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:11.381088    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"
Nov 01 06:12:11.381220 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:11.381117    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7" event={"ID":"e59595a9-a730-43f0-a4cd-c667b99f57d6","Type":"ContainerDied","Data":"93c2f3fe9d5c55bb8793e0622633a7d82aa5e47e03aca034c43b28bed7287eef"}
Nov 01 06:12:11.381220 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:11.381172    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7" event={"ID":"e59595a9-a730-43f0-a4cd-c667b99f57d6","Type":"ContainerDied","Data":"c14f7598248d7139208a8c6b5d4ff2917a13f096ae81ed467750a9a1a364d807"}
Nov 01 06:12:11.381220 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:11.381195    5619 scope.go:122] "RemoveContainer" containerID="93c2f3fe9d5c55bb8793e0622633a7d82aa5e47e03aca034c43b28bed7287eef"
Nov 01 06:12:11.391309 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:11.391290    5619 scope.go:122] "RemoveContainer" containerID="93c2f3fe9d5c55bb8793e0622633a7d82aa5e47e03aca034c43b28bed7287eef"
Nov 01 06:12:11.391654 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:12:11.391630    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c2f3fe9d5c55bb8793e0622633a7d82aa5e47e03aca034c43b28bed7287eef\": container with ID starting with 93c2f3fe9d5c55bb8793e0622633a7d82aa5e47e03aca034c43b28bed7287eef not found: ID does not exist" containerID="93c2f3fe9d5c55bb8793e0622633a7d82aa5e47e03aca034c43b28bed7287eef"
Nov 01 06:12:11.391704 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:11.391665    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c2f3fe9d5c55bb8793e0622633a7d82aa5e47e03aca034c43b28bed7287eef"} err="failed to get container status \"93c2f3fe9d5c55bb8793e0622633a7d82aa5e47e03aca034c43b28bed7287eef\": rpc error: code = NotFound desc = could not find container \"93c2f3fe9d5c55bb8793e0622633a7d82aa5e47e03aca034c43b28bed7287eef\": container with ID starting with 93c2f3fe9d5c55bb8793e0622633a7d82aa5e47e03aca034c43b28bed7287eef not found: ID does not exist"
Nov 01 06:12:11.574671 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:11.574637    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"]
Nov 01 06:12:11.624937 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:11.624908    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77htgc7"]
Nov 01 06:12:11.895287 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:11.895246    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:12.308871 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:12.308796    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq"]
Nov 01 06:12:12.309046 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:12.309017    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq" podUID="8a586fea-87b6-4599-beed-36aba543e3c9" containerName="payload-pre-processing" containerID="cri-o://cf82a2b426f0540c81393b87b754f301d30e18651a20870676b7824dc59b6a21" gracePeriod=30
Nov 01 06:12:12.403405 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:12.403368    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:12.598753 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:12.598727    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:12.599587 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:12.599570    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq"
Nov 01 06:12:12.621048 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:12.621028    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a586fea-87b6-4599-beed-36aba543e3c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a586fea-87b6-4599-beed-36aba543e3c9-plugins-config-volume\") pod \"8a586fea-87b6-4599-beed-36aba543e3c9\" (UID: \"8a586fea-87b6-4599-beed-36aba543e3c9\") "
Nov 01 06:12:12.621132 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:12.621080    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a586fea-87b6-4599-beed-36aba543e3c9-kube-api-access-8qjkd\" (UniqueName: \"kubernetes.io/projected/8a586fea-87b6-4599-beed-36aba543e3c9-kube-api-access-8qjkd\") pod \"8a586fea-87b6-4599-beed-36aba543e3c9\" (UID: \"8a586fea-87b6-4599-beed-36aba543e3c9\") "
Nov 01 06:12:12.621418 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:12.621395    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a586fea-87b6-4599-beed-36aba543e3c9-plugins-config-volume" pod "8a586fea-87b6-4599-beed-36aba543e3c9" (UID: "8a586fea-87b6-4599-beed-36aba543e3c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:12:12.621611 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:12.621597    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a586fea-87b6-4599-beed-36aba543e3c9-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:12.623052 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:12.623034    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a586fea-87b6-4599-beed-36aba543e3c9-kube-api-access-8qjkd" pod "8a586fea-87b6-4599-beed-36aba543e3c9" (UID: "8a586fea-87b6-4599-beed-36aba543e3c9"). InnerVolumeSpecName "kube-api-access-8qjkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:12:12.723227 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:12.723190    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qjkd\" (UniqueName: \"kubernetes.io/projected/8a586fea-87b6-4599-beed-36aba543e3c9-kube-api-access-8qjkd\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:12.964563 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:12.964533    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e59595a9-a730-43f0-a4cd-c667b99f57d6" path="/var/lib/kubelet/pods/e59595a9-a730-43f0-a4cd-c667b99f57d6/volumes"
Nov 01 06:12:13.396652 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:13.396621    5619 generic.go:356] "Generic (PLEG): container finished" podID="8a586fea-87b6-4599-beed-36aba543e3c9" containerID="cf82a2b426f0540c81393b87b754f301d30e18651a20870676b7824dc59b6a21" exitCode=0
Nov 01 06:12:13.396854 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:13.396680    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq"
Nov 01 06:12:13.396854 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:13.396707    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq" event={"ID":"8a586fea-87b6-4599-beed-36aba543e3c9","Type":"ContainerDied","Data":"cf82a2b426f0540c81393b87b754f301d30e18651a20870676b7824dc59b6a21"}
Nov 01 06:12:13.396854 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:13.396764    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq" event={"ID":"8a586fea-87b6-4599-beed-36aba543e3c9","Type":"ContainerDied","Data":"5922c421e8902e316db3e27d96515a5fec981432315295ee01d9c248fce0f659"}
Nov 01 06:12:13.396854 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:13.396787    5619 scope.go:122] "RemoveContainer" containerID="cf82a2b426f0540c81393b87b754f301d30e18651a20870676b7824dc59b6a21"
Nov 01 06:12:13.407504 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:13.407481    5619 scope.go:122] "RemoveContainer" containerID="cf82a2b426f0540c81393b87b754f301d30e18651a20870676b7824dc59b6a21"
Nov 01 06:12:13.407770 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:12:13.407750    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf82a2b426f0540c81393b87b754f301d30e18651a20870676b7824dc59b6a21\": container with ID starting with cf82a2b426f0540c81393b87b754f301d30e18651a20870676b7824dc59b6a21 not found: ID does not exist" containerID="cf82a2b426f0540c81393b87b754f301d30e18651a20870676b7824dc59b6a21"
Nov 01 06:12:13.407833 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:13.407779    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf82a2b426f0540c81393b87b754f301d30e18651a20870676b7824dc59b6a21"} err="failed to get container status \"cf82a2b426f0540c81393b87b754f301d30e18651a20870676b7824dc59b6a21\": rpc error: code = NotFound desc = could not find container \"cf82a2b426f0540c81393b87b754f301d30e18651a20870676b7824dc59b6a21\": container with ID starting with cf82a2b426f0540c81393b87b754f301d30e18651a20870676b7824dc59b6a21 not found: ID does not exist"
Nov 01 06:12:13.584356 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:13.583499    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq"]
Nov 01 06:12:13.633389 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:13.633353    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d3f8d-775866ccf9-2sznq"]
Nov 01 06:12:13.699030 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:13.698941    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:14.963494 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:14.963467    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a586fea-87b6-4599-beed-36aba543e3c9" path="/var/lib/kubelet/pods/8a586fea-87b6-4599-beed-36aba543e3c9/volumes"
Nov 01 06:12:15.110917 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:15.110880    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:15.251266 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:15.251172    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:16.670821 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:16.670783    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:17.160931 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:17.160888    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:17.177193 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:17.176538    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"]
Nov 01 06:12:17.177732 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:17.177701    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn" podUID="e5087163-8f0a-407b-b2be-cb9f9c888706" containerName="istio-proxy" containerID="cri-o://791ac794f7c6cd9238eb82e307ef868cf6e5bd53c285ad4a6ce170229b4459d0" gracePeriod=30
Nov 01 06:12:18.721230 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:18.721193    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:20.271607 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:20.271562    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:22.488614 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.488586    5619 generic.go:356] "Generic (PLEG): container finished" podID="e5087163-8f0a-407b-b2be-cb9f9c888706" containerID="791ac794f7c6cd9238eb82e307ef868cf6e5bd53c285ad4a6ce170229b4459d0" exitCode=0
Nov 01 06:12:22.489035 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.488654    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn" event={"ID":"e5087163-8f0a-407b-b2be-cb9f9c888706","Type":"ContainerDied","Data":"791ac794f7c6cd9238eb82e307ef868cf6e5bd53c285ad4a6ce170229b4459d0"}
Nov 01 06:12:22.489035 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.488708    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn" event={"ID":"e5087163-8f0a-407b-b2be-cb9f9c888706","Type":"ContainerDied","Data":"0ad740fe6381ee32370c62b9de6208111d558690631d614ddaed3c7dd19f606a"}
Nov 01 06:12:22.489035 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.488725    5619 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ad740fe6381ee32370c62b9de6208111d558690631d614ddaed3c7dd19f606a"
Nov 01 06:12:22.489337 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.489310    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:12:22.578265 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.578219    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-envoy\") pod \"e5087163-8f0a-407b-b2be-cb9f9c888706\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") "
Nov 01 06:12:22.578498 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.578275    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-workload-certs\") pod \"e5087163-8f0a-407b-b2be-cb9f9c888706\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") "
Nov 01 06:12:22.578498 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.578311    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-podinfo\") pod \"e5087163-8f0a-407b-b2be-cb9f9c888706\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") "
Nov 01 06:12:22.578498 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.578376    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-data\") pod \"e5087163-8f0a-407b-b2be-cb9f9c888706\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") "
Nov 01 06:12:22.578498 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.578424    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-credential-socket\") pod \"e5087163-8f0a-407b-b2be-cb9f9c888706\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") "
Nov 01 06:12:22.578763 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.578573    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-workload-certs" pod "e5087163-8f0a-407b-b2be-cb9f9c888706" (UID: "e5087163-8f0a-407b-b2be-cb9f9c888706"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:12:22.578763 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.578583    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5087163-8f0a-407b-b2be-cb9f9c888706-kube-api-access-rgnld\" (UniqueName: \"kubernetes.io/projected/e5087163-8f0a-407b-b2be-cb9f9c888706-kube-api-access-rgnld\") pod \"e5087163-8f0a-407b-b2be-cb9f9c888706\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") "
Nov 01 06:12:22.578763 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.578656    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-token\" (UniqueName: \"kubernetes.io/projected/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-token\") pod \"e5087163-8f0a-407b-b2be-cb9f9c888706\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") "
Nov 01 06:12:22.578921 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.578778    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5087163-8f0a-407b-b2be-cb9f9c888706-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5087163-8f0a-407b-b2be-cb9f9c888706-istiod-ca-cert\") pod \"e5087163-8f0a-407b-b2be-cb9f9c888706\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") "
Nov 01 06:12:22.578921 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.578794    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-data" pod "e5087163-8f0a-407b-b2be-cb9f9c888706" (UID: "e5087163-8f0a-407b-b2be-cb9f9c888706"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:12:22.578921 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.578877    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-workload-socket\") pod \"e5087163-8f0a-407b-b2be-cb9f9c888706\" (UID: \"e5087163-8f0a-407b-b2be-cb9f9c888706\") "
Nov 01 06:12:22.579477 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.579453    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:22.579593 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.579482    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:22.579668 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.579646    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-credential-socket" pod "e5087163-8f0a-407b-b2be-cb9f9c888706" (UID: "e5087163-8f0a-407b-b2be-cb9f9c888706"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:12:22.579668 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.579657    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-workload-socket" pod "e5087163-8f0a-407b-b2be-cb9f9c888706" (UID: "e5087163-8f0a-407b-b2be-cb9f9c888706"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:12:22.579813 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.579780    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5087163-8f0a-407b-b2be-cb9f9c888706-istiod-ca-cert" pod "e5087163-8f0a-407b-b2be-cb9f9c888706" (UID: "e5087163-8f0a-407b-b2be-cb9f9c888706"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:12:22.582304 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.582264    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-podinfo" pod "e5087163-8f0a-407b-b2be-cb9f9c888706" (UID: "e5087163-8f0a-407b-b2be-cb9f9c888706"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:12:22.584867 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.584846    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5087163-8f0a-407b-b2be-cb9f9c888706-kube-api-access-rgnld" pod "e5087163-8f0a-407b-b2be-cb9f9c888706" (UID: "e5087163-8f0a-407b-b2be-cb9f9c888706"). InnerVolumeSpecName "kube-api-access-rgnld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:12:22.585481 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.585443    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-envoy" pod "e5087163-8f0a-407b-b2be-cb9f9c888706" (UID: "e5087163-8f0a-407b-b2be-cb9f9c888706"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:12:22.585805 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.585780    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-token" pod "e5087163-8f0a-407b-b2be-cb9f9c888706" (UID: "e5087163-8f0a-407b-b2be-cb9f9c888706"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:12:22.680851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.680814    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgnld\" (UniqueName: \"kubernetes.io/projected/e5087163-8f0a-407b-b2be-cb9f9c888706-kube-api-access-rgnld\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:22.680851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.680855    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:22.681059 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.680871    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5087163-8f0a-407b-b2be-cb9f9c888706-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:22.681059 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.680886    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:22.681059 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.680901    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:22.681059 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.680916    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5087163-8f0a-407b-b2be-cb9f9c888706-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:22.681059 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:22.680929    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5087163-8f0a-407b-b2be-cb9f9c888706-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:23.495367 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:23.495314    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"
Nov 01 06:12:23.677597 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:23.677558    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:23.718354 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:23.718287    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"]
Nov 01 06:12:23.766726 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:23.766623    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d3f8d-openshift-default-6b96ff55c5-gvkgn"]
Nov 01 06:12:24.962687 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:24.962655    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5087163-8f0a-407b-b2be-cb9f9c888706" path="/var/lib/kubelet/pods/e5087163-8f0a-407b-b2be-cb9f9c888706/volumes"
Nov 01 06:12:25.139749 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:25.139713    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:25.879809 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:12:25.879774    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:12:28.474479 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:28.474439    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:29.537266 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:29.537230    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"]
Nov 01 06:12:32.802224 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:32.802186    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:33.339394 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:33.339359    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:34.538379 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:34.538306    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq" podUID="add83dd5-a128-499e-9e8b-00142cc7bc94" containerName="payload-processing" containerID="cri-o://6bb73475d0f0e8a130b118beaab7918ff1008f83e33e8325db30469bfaf657db" gracePeriod=25
Nov 01 06:12:34.799654 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:34.799592    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:12:34.830047 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:34.830016    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add83dd5-a128-499e-9e8b-00142cc7bc94-provider-credentials\" (UniqueName: \"kubernetes.io/projected/add83dd5-a128-499e-9e8b-00142cc7bc94-provider-credentials\") pod \"add83dd5-a128-499e-9e8b-00142cc7bc94\" (UID: \"add83dd5-a128-499e-9e8b-00142cc7bc94\") "
Nov 01 06:12:34.830229 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:34.830213    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/add83dd5-a128-499e-9e8b-00142cc7bc94-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/add83dd5-a128-499e-9e8b-00142cc7bc94-routing-overlay-volume\") pod \"add83dd5-a128-499e-9e8b-00142cc7bc94\" (UID: \"add83dd5-a128-499e-9e8b-00142cc7bc94\") "
Nov 01 06:12:34.830346 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:34.830318    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add83dd5-a128-499e-9e8b-00142cc7bc94-kube-api-access-668j4\" (UniqueName: \"kubernetes.io/projected/add83dd5-a128-499e-9e8b-00142cc7bc94-kube-api-access-668j4\") pod \"add83dd5-a128-499e-9e8b-00142cc7bc94\" (UID: \"add83dd5-a128-499e-9e8b-00142cc7bc94\") "
Nov 01 06:12:34.830437 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:34.830424    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/add83dd5-a128-499e-9e8b-00142cc7bc94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add83dd5-a128-499e-9e8b-00142cc7bc94-plugins-config-volume\") pod \"add83dd5-a128-499e-9e8b-00142cc7bc94\" (UID: \"add83dd5-a128-499e-9e8b-00142cc7bc94\") "
Nov 01 06:12:34.830720 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:34.830593    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add83dd5-a128-499e-9e8b-00142cc7bc94-routing-overlay-volume" pod "add83dd5-a128-499e-9e8b-00142cc7bc94" (UID: "add83dd5-a128-499e-9e8b-00142cc7bc94"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:12:34.830853 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:34.830803    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add83dd5-a128-499e-9e8b-00142cc7bc94-plugins-config-volume" pod "add83dd5-a128-499e-9e8b-00142cc7bc94" (UID: "add83dd5-a128-499e-9e8b-00142cc7bc94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:12:34.830959 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:34.830938    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/add83dd5-a128-499e-9e8b-00142cc7bc94-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:34.831032 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:34.830964    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add83dd5-a128-499e-9e8b-00142cc7bc94-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:34.832278 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:34.832255    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add83dd5-a128-499e-9e8b-00142cc7bc94-provider-credentials" pod "add83dd5-a128-499e-9e8b-00142cc7bc94" (UID: "add83dd5-a128-499e-9e8b-00142cc7bc94"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:12:34.832476 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:34.832455    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add83dd5-a128-499e-9e8b-00142cc7bc94-kube-api-access-668j4" pod "add83dd5-a128-499e-9e8b-00142cc7bc94" (UID: "add83dd5-a128-499e-9e8b-00142cc7bc94"). InnerVolumeSpecName "kube-api-access-668j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:12:34.932271 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:34.932240    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-668j4\" (UniqueName: \"kubernetes.io/projected/add83dd5-a128-499e-9e8b-00142cc7bc94-kube-api-access-668j4\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:34.932271 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:34.932269    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/add83dd5-a128-499e-9e8b-00142cc7bc94-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:12:35.572838 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:35.572801    5619 generic.go:356] "Generic (PLEG): container finished" podID="add83dd5-a128-499e-9e8b-00142cc7bc94" containerID="6bb73475d0f0e8a130b118beaab7918ff1008f83e33e8325db30469bfaf657db" exitCode=0
Nov 01 06:12:35.573282 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:35.572848    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"
Nov 01 06:12:35.573282 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:35.572883    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq" event={"ID":"add83dd5-a128-499e-9e8b-00142cc7bc94","Type":"ContainerDied","Data":"6bb73475d0f0e8a130b118beaab7918ff1008f83e33e8325db30469bfaf657db"}
Nov 01 06:12:35.573282 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:35.572963    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq" event={"ID":"add83dd5-a128-499e-9e8b-00142cc7bc94","Type":"ContainerDied","Data":"e3c035b2c313506e728379d8502473821b6bc4604c481755707241507abbf9e9"}
Nov 01 06:12:35.573282 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:35.573000    5619 scope.go:122] "RemoveContainer" containerID="6bb73475d0f0e8a130b118beaab7918ff1008f83e33e8325db30469bfaf657db"
Nov 01 06:12:35.584541 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:35.584523    5619 scope.go:122] "RemoveContainer" containerID="6bb73475d0f0e8a130b118beaab7918ff1008f83e33e8325db30469bfaf657db"
Nov 01 06:12:35.584832 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:12:35.584810    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb73475d0f0e8a130b118beaab7918ff1008f83e33e8325db30469bfaf657db\": container with ID starting with 6bb73475d0f0e8a130b118beaab7918ff1008f83e33e8325db30469bfaf657db not found: ID does not exist" containerID="6bb73475d0f0e8a130b118beaab7918ff1008f83e33e8325db30469bfaf657db"
Nov 01 06:12:35.584940 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:35.584846    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb73475d0f0e8a130b118beaab7918ff1008f83e33e8325db30469bfaf657db"} err="failed to get container status \"6bb73475d0f0e8a130b118beaab7918ff1008f83e33e8325db30469bfaf657db\": rpc error: code = NotFound desc = could not find container \"6bb73475d0f0e8a130b118beaab7918ff1008f83e33e8325db30469bfaf657db\": container with ID starting with 6bb73475d0f0e8a130b118beaab7918ff1008f83e33e8325db30469bfaf657db not found: ID does not exist"
Nov 01 06:12:35.740990 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:35.740946    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"]
Nov 01 06:12:35.791777 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:35.791746    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b979wkrxq"]
Nov 01 06:12:36.963551 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:36.963521    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="add83dd5-a128-499e-9e8b-00142cc7bc94" path="/var/lib/kubelet/pods/add83dd5-a128-499e-9e8b-00142cc7bc94/volumes"
Nov 01 06:12:38.222080 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:38.222039    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:38.382371 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:38.382302    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:39.393215 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:39.393169    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:39.542845 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:39.542811    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:39.690511 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:39.690424    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:40.003689 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:40.003603    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:40.659834 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:40.659804    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:40.855386 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:40.855317    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:43.299951 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:43.299915    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:48.098830 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:48.098793    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:49.840677 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:49.840633    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:49.986870 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:49.986823    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:50.840936 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:50.840896    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:51.034581 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:51.034549    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:52.939385 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:52.935270    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:55.021993 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:55.021961    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:55.879868 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:12:55.879832    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:12:57.014745 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:57.014713    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:57.465596 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:57.465553    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:57.618165 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:57.617958    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:12:57.774482 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:12:57.774400    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:02.567499 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:13:02.567444    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:02.780806 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:13:02.780770    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:03.229164 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:13:03.229125    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:03.237271 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:13:03.237246    5619 scope.go:122] "RemoveContainer" containerID="791ac794f7c6cd9238eb82e307ef868cf6e5bd53c285ad4a6ce170229b4459d0"
Nov 01 06:13:07.387272 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:13:07.387226    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:12.202831 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:13:12.202739    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:16.964236 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:13:16.964199    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:20.663547 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:13:20.663511    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:21.758745 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:13:21.758704    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:25.519385 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:13:25.519339    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:25.888523 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:13:25.888483    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:13:26.559932 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:13:26.559895    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:29.820118 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:13:29.820081    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:31.366130 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:13:31.366081    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:36.170224 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:13:36.170188    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:40.963126 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:13:40.963081    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:45.759903 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:13:45.759865    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:50.474491 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:13:50.474458    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:55.278040 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:13:55.273160    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:13:55.879230 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:13:55.879192    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:14:00.068362 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:14:00.068297    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:04.848847 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:14:04.848810    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:09.570569 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:14:09.570536    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:14.357460 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:14:14.357422    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:19.151978 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:14:19.151929    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:21.009672 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:14:21.009640    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:23.984054 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:14:23.984017    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:25.833219 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:14:25.833187    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:25.880201 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:14:25.880167    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:14:27.144596 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:14:27.144477    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:27.287290 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:14:27.287218    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:28.769528 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:14:28.769492    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:33.567601 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:14:33.567565    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:35.782690 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:14:35.782644    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:38.020795 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:14:38.020760    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:38.314982 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:14:38.314894    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:40.833701 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:14:40.833608    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:43.076755 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:14:43.076718    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:47.956765 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:14:47.956727    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:52.769863 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:14:52.769814    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:14:55.881291 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:14:55.881248    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:14:57.570698 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:14:57.570652    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:02.468926 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:02.468886    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:07.267947 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:07.267906    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:09.067239 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.067203    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"]
Nov 01 06:15:09.067789 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.067767    5619 cpu_manager.go:414] "containerMap: removing container" podUID="e5087163-8f0a-407b-b2be-cb9f9c888706" containerName="istio-proxy"
Nov 01 06:15:09.067789 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.067783    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5087163-8f0a-407b-b2be-cb9f9c888706" containerName="istio-proxy"
Nov 01 06:15:09.067887 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.067805    5619 cpu_manager.go:414] "containerMap: removing container" podUID="e59595a9-a730-43f0-a4cd-c667b99f57d6" containerName="payload-processing"
Nov 01 06:15:09.067887 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.067813    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e59595a9-a730-43f0-a4cd-c667b99f57d6" containerName="payload-processing"
Nov 01 06:15:09.067887 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.067834    5619 cpu_manager.go:414] "containerMap: removing container" podUID="add83dd5-a128-499e-9e8b-00142cc7bc94" containerName="payload-processing"
Nov 01 06:15:09.067887 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.067842    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add83dd5-a128-499e-9e8b-00142cc7bc94" containerName="payload-processing"
Nov 01 06:15:09.067887 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.067869    5619 cpu_manager.go:414] "containerMap: removing container" podUID="8a586fea-87b6-4599-beed-36aba543e3c9" containerName="payload-pre-processing"
Nov 01 06:15:09.067887 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.067875    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a586fea-87b6-4599-beed-36aba543e3c9" containerName="payload-pre-processing"
Nov 01 06:15:09.068081 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.067980    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="e59595a9-a730-43f0-a4cd-c667b99f57d6" containerName="payload-processing"
Nov 01 06:15:09.068081 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.067989    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a586fea-87b6-4599-beed-36aba543e3c9" containerName="payload-pre-processing"
Nov 01 06:15:09.068081 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.068000    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="add83dd5-a128-499e-9e8b-00142cc7bc94" containerName="payload-processing"
Nov 01 06:15:09.068081 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.068009    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5087163-8f0a-407b-b2be-cb9f9c888706" containerName="istio-proxy"
Nov 01 06:15:09.076766 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.076739    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.130359 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.130312    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9c8fc5-openshift-default-dockercfg-6mn84\""
Nov 01 06:15:09.167410 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.167372    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.167591 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.167427    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44c28d4-b7ca-4350-b78a-53e4c9036124-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.167591 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.167488    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.167705 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.167606    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.167705 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.167671    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxk9v\" (UniqueName: \"kubernetes.io/projected/b44c28d4-b7ca-4350-b78a-53e4c9036124-kube-api-access-mxk9v\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.167808 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.167789    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.167866 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.167851    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.167938 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.167919    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.168010 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.167992    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.184583 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.184554    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"]
Nov 01 06:15:09.269184 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.269149    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.269400 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.269208    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.269400 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.269266    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxk9v\" (UniqueName: \"kubernetes.io/projected/b44c28d4-b7ca-4350-b78a-53e4c9036124-kube-api-access-mxk9v\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.269400 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.269367    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.269580 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.269444    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.269580 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.269544    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.269688 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.269606    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.269688 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.269614    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.269688 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.269637    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.269797 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.269752    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.269797 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.269782    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44c28d4-b7ca-4350-b78a-53e4c9036124-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.270132 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.270110    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.270231 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.270171    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.270492 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.270472    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44c28d4-b7ca-4350-b78a-53e4c9036124-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.271647 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.271627    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.271930 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.271911    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.279133 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.279111    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"]
Nov 01 06:15:09.279381 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.279343    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs" podUID="074ba060-2a84-479d-9d8b-92987fb010e9" containerName="istio-proxy" containerID="cri-o://bb33628316f779c952e16a48ffe64795df08196c9b221cad5055fcc016b56ec7" gracePeriod=30
Nov 01 06:15:09.282205 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.282173    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"]
Nov 01 06:15:09.282469 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.282424    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7" podUID="231e12b3-f2c4-4633-8a9b-4dcfc8c34206" containerName="istio-proxy" containerID="cri-o://6e307a6f756f4ca43804d8d82e122eca5705ab8619e430f43d4cac8d515e6331" gracePeriod=30
Nov 01 06:15:09.286054 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.286030    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"]
Nov 01 06:15:09.286312 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.286268    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9" podUID="e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" containerName="istio-proxy" containerID="cri-o://5aeb4125a45b0f389c81cc59878c22a41cef1e64aab13fa696d7240abe4802bd" gracePeriod=30
Nov 01 06:15:09.331994 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.331906    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.333159 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.333123    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxk9v\" (UniqueName: \"kubernetes.io/projected/b44c28d4-b7ca-4350-b78a-53e4c9036124-kube-api-access-mxk9v\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.390957 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.390926    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:09.832282 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.832248    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"]
Nov 01 06:15:09.833107 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:15:09.833077    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44c28d4_b7ca_4350_b78a_53e4c9036124.slice/crio-8d35e306228c017abb5f458818542431d0bb3938434d040e156792aef5b089c7 WatchSource:0}: Error finding container 8d35e306228c017abb5f458818542431d0bb3938434d040e156792aef5b089c7: Status 404 returned error can't find the container with id 8d35e306228c017abb5f458818542431d0bb3938434d040e156792aef5b089c7
Nov 01 06:15:09.835492 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.835462    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:15:09.835555 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.835525    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:15:09.835592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:09.835555    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:15:10.530536 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:10.530498    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm" event={"ID":"b44c28d4-b7ca-4350-b78a-53e4c9036124","Type":"ContainerStarted","Data":"922fea25a0e6087c22d7893da1c361252c89c5db7f3fa0d25b33306e81f25b25"}
Nov 01 06:15:10.530992 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:10.530565    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm" event={"ID":"b44c28d4-b7ca-4350-b78a-53e4c9036124","Type":"ContainerStarted","Data":"8d35e306228c017abb5f458818542431d0bb3938434d040e156792aef5b089c7"}
Nov 01 06:15:10.698883 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:10.698811    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm" podStartSLOduration=1.6987958060000001 podStartE2EDuration="1.698795806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:10.697660964 +0000 UTC m=+1747.908484473" watchObservedRunningTime="2026-11-01 06:15:10.698795806 +0000 UTC m=+1747.909619314"
Nov 01 06:15:11.391546 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:11.391509    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:11.393002 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:11.392977    5619 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 01 06:15:11.393136 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:11.393050    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm" podUID="b44c28d4-b7ca-4350-b78a-53e4c9036124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 01 06:15:12.201788 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:12.201758    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:12.392552 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:12.391862    5619 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 01 06:15:12.392552 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:12.391924    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm" podUID="b44c28d4-b7ca-4350-b78a-53e4c9036124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 01 06:15:13.392012 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:13.391979    5619 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 01 06:15:13.392498 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:13.392030    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm" podUID="b44c28d4-b7ca-4350-b78a-53e4c9036124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 01 06:15:14.393000 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.392970    5619 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 01 06:15:14.393419 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.393383    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm" podUID="b44c28d4-b7ca-4350-b78a-53e4c9036124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 01 06:15:14.589160 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.589076    5619 generic.go:356] "Generic (PLEG): container finished" podID="e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" containerID="5aeb4125a45b0f389c81cc59878c22a41cef1e64aab13fa696d7240abe4802bd" exitCode=0
Nov 01 06:15:14.590072 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.589373    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9" event={"ID":"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163","Type":"ContainerDied","Data":"5aeb4125a45b0f389c81cc59878c22a41cef1e64aab13fa696d7240abe4802bd"}
Nov 01 06:15:14.593367 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.593181    5619 generic.go:356] "Generic (PLEG): container finished" podID="074ba060-2a84-479d-9d8b-92987fb010e9" containerID="bb33628316f779c952e16a48ffe64795df08196c9b221cad5055fcc016b56ec7" exitCode=0
Nov 01 06:15:14.593367 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.593259    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs" event={"ID":"074ba060-2a84-479d-9d8b-92987fb010e9","Type":"ContainerDied","Data":"bb33628316f779c952e16a48ffe64795df08196c9b221cad5055fcc016b56ec7"}
Nov 01 06:15:14.596115 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.595632    5619 generic.go:356] "Generic (PLEG): container finished" podID="231e12b3-f2c4-4633-8a9b-4dcfc8c34206" containerID="6e307a6f756f4ca43804d8d82e122eca5705ab8619e430f43d4cac8d515e6331" exitCode=0
Nov 01 06:15:14.596115 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.595710    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7" event={"ID":"231e12b3-f2c4-4633-8a9b-4dcfc8c34206","Type":"ContainerDied","Data":"6e307a6f756f4ca43804d8d82e122eca5705ab8619e430f43d4cac8d515e6331"}
Nov 01 06:15:14.621970 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.621939    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:15:14.739941 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.739911    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-envoy\") pod \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") "
Nov 01 06:15:14.740082 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.739978    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-workload-socket\") pod \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") "
Nov 01 06:15:14.740148 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.740113    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-data\") pod \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") "
Nov 01 06:15:14.740199 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.740151    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-workload-certs\") pod \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") "
Nov 01 06:15:14.740199 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.740181    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-token\" (UniqueName: \"kubernetes.io/projected/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-token\") pod \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") "
Nov 01 06:15:14.740267 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.740252    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-credential-socket\") pod \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") "
Nov 01 06:15:14.740301 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.740278    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istiod-ca-cert\") pod \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") "
Nov 01 06:15:14.740364 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.740356    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-kube-api-access-p64nr\" (UniqueName: \"kubernetes.io/projected/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-kube-api-access-p64nr\") pod \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") "
Nov 01 06:15:14.740405 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.740375    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-podinfo\") pod \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\" (UID: \"231e12b3-f2c4-4633-8a9b-4dcfc8c34206\") "
Nov 01 06:15:14.741711 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.741683    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-data" pod "231e12b3-f2c4-4633-8a9b-4dcfc8c34206" (UID: "231e12b3-f2c4-4633-8a9b-4dcfc8c34206"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:15:14.741818 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.741697    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-workload-socket" pod "231e12b3-f2c4-4633-8a9b-4dcfc8c34206" (UID: "231e12b3-f2c4-4633-8a9b-4dcfc8c34206"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:15:14.741818 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.741775    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:15:14.741940 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.741905    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-workload-certs" pod "231e12b3-f2c4-4633-8a9b-4dcfc8c34206" (UID: "231e12b3-f2c4-4633-8a9b-4dcfc8c34206"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:15:14.742081 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.742065    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-credential-socket" pod "231e12b3-f2c4-4633-8a9b-4dcfc8c34206" (UID: "231e12b3-f2c4-4633-8a9b-4dcfc8c34206"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:15:14.742166 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.742122    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istiod-ca-cert" pod "231e12b3-f2c4-4633-8a9b-4dcfc8c34206" (UID: "231e12b3-f2c4-4633-8a9b-4dcfc8c34206"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:15:14.742166 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.742135    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-envoy" pod "231e12b3-f2c4-4633-8a9b-4dcfc8c34206" (UID: "231e12b3-f2c4-4633-8a9b-4dcfc8c34206"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:15:14.743411 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.743375    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-podinfo" pod "231e12b3-f2c4-4633-8a9b-4dcfc8c34206" (UID: "231e12b3-f2c4-4633-8a9b-4dcfc8c34206"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:15:14.744171 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.744148    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-kube-api-access-p64nr" pod "231e12b3-f2c4-4633-8a9b-4dcfc8c34206" (UID: "231e12b3-f2c4-4633-8a9b-4dcfc8c34206"). InnerVolumeSpecName "kube-api-access-p64nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:15:14.745005 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.744979    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-token" pod "231e12b3-f2c4-4633-8a9b-4dcfc8c34206" (UID: "231e12b3-f2c4-4633-8a9b-4dcfc8c34206"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:15:14.747025 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.746971    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 06:15:14.841469 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.841440    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-credential-socket\") pod \"074ba060-2a84-479d-9d8b-92987fb010e9\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") "
Nov 01 06:15:14.841677 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.841484    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-data\") pod \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") "
Nov 01 06:15:14.841677 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.841551    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-kube-api-access-wchpg\" (UniqueName: \"kubernetes.io/projected/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-kube-api-access-wchpg\") pod \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") "
Nov 01 06:15:14.841677 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.841591    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-workload-socket\") pod \"074ba060-2a84-479d-9d8b-92987fb010e9\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") "
Nov 01 06:15:14.841859 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.841758    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/074ba060-2a84-479d-9d8b-92987fb010e9-kube-api-access-gzrqq\" (UniqueName: \"kubernetes.io/projected/074ba060-2a84-479d-9d8b-92987fb010e9-kube-api-access-gzrqq\") pod \"074ba060-2a84-479d-9d8b-92987fb010e9\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") "
Nov 01 06:15:14.841859 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.841819    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-envoy\") pod \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") "
Nov 01 06:15:14.841954 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.841863    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-data" pod "e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" (UID: "e17e1eaf-fae0-49af-bfe9-8fe6b04bc163"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:15:14.841954 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.841877    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-workload-socket" pod "074ba060-2a84-479d-9d8b-92987fb010e9" (UID: "074ba060-2a84-479d-9d8b-92987fb010e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:15:14.841954 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.841920    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istiod-ca-cert\") pod \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") "
Nov 01 06:15:14.842125 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.841980    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-credential-socket\") pod \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") "
Nov 01 06:15:14.842125 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.842038    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-workload-certs\") pod \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") "
Nov 01 06:15:14.842125 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.842088    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-workload-socket\") pod \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") "
Nov 01 06:15:14.842274 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.842131    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-token\" (UniqueName: \"kubernetes.io/projected/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-token\") pod \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") "
Nov 01 06:15:14.842274 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.842160    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-podinfo\") pod \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\" (UID: \"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163\") "
Nov 01 06:15:14.842405 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.842308    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-credential-socket" pod "074ba060-2a84-479d-9d8b-92987fb010e9" (UID: "074ba060-2a84-479d-9d8b-92987fb010e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:15:14.842464 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.842440    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-workload-certs" pod "e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" (UID: "e17e1eaf-fae0-49af-bfe9-8fe6b04bc163"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:15:14.842563 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.842544    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-istio-envoy\") pod \"074ba060-2a84-479d-9d8b-92987fb010e9\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") "
Nov 01 06:15:14.842620 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.842602    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/074ba060-2a84-479d-9d8b-92987fb010e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/074ba060-2a84-479d-9d8b-92987fb010e9-istio-podinfo\") pod \"074ba060-2a84-479d-9d8b-92987fb010e9\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") "
Nov 01 06:15:14.842620 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.842609    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istiod-ca-cert" pod "e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" (UID: "e17e1eaf-fae0-49af-bfe9-8fe6b04bc163"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:15:14.842736 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.842634    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-workload-socket" pod "e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" (UID: "e17e1eaf-fae0-49af-bfe9-8fe6b04bc163"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:15:14.842736 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.842693    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/074ba060-2a84-479d-9d8b-92987fb010e9-istio-token\" (UniqueName: \"kubernetes.io/projected/074ba060-2a84-479d-9d8b-92987fb010e9-istio-token\") pod \"074ba060-2a84-479d-9d8b-92987fb010e9\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") "
Nov 01 06:15:14.842851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.842743    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-workload-certs\") pod \"074ba060-2a84-479d-9d8b-92987fb010e9\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") "
Nov 01 06:15:14.842851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.842776    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/074ba060-2a84-479d-9d8b-92987fb010e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/074ba060-2a84-479d-9d8b-92987fb010e9-istiod-ca-cert\") pod \"074ba060-2a84-479d-9d8b-92987fb010e9\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") "
Nov 01 06:15:14.842851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.842833    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-istio-data\") pod \"074ba060-2a84-479d-9d8b-92987fb010e9\" (UID: \"074ba060-2a84-479d-9d8b-92987fb010e9\") "
Nov 01 06:15:14.843474 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.843449    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p64nr\" (UniqueName: \"kubernetes.io/projected/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-kube-api-access-p64nr\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.843573 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.843481    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.843573 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.843498    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.843573 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.843514    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.843573 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.843531    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.843573 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.843548    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.843573 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.843563    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.843886 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.843577    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.843886 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.843593    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.843886 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.843789    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-credential-socket" pod "e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" (UID: "e17e1eaf-fae0-49af-bfe9-8fe6b04bc163"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:15:14.844089 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.843608    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.844156 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.844102    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.844156 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.844121    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.844156 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.844139    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.844334 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.844154    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.844334 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.844169    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231e12b3-f2c4-4633-8a9b-4dcfc8c34206-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.844461 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.844406    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-istio-data" pod "074ba060-2a84-479d-9d8b-92987fb010e9" (UID: "074ba060-2a84-479d-9d8b-92987fb010e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:15:14.844615 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.844590    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-workload-certs" pod "074ba060-2a84-479d-9d8b-92987fb010e9" (UID: "074ba060-2a84-479d-9d8b-92987fb010e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:15:14.844945 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.844921    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074ba060-2a84-479d-9d8b-92987fb010e9-istiod-ca-cert" pod "074ba060-2a84-479d-9d8b-92987fb010e9" (UID: "074ba060-2a84-479d-9d8b-92987fb010e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:15:14.846357 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.846299    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-token" pod "e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" (UID: "e17e1eaf-fae0-49af-bfe9-8fe6b04bc163"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:15:14.847050 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.847020    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-podinfo" pod "e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" (UID: "e17e1eaf-fae0-49af-bfe9-8fe6b04bc163"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:15:14.847050 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.847035    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074ba060-2a84-479d-9d8b-92987fb010e9-kube-api-access-gzrqq" pod "074ba060-2a84-479d-9d8b-92987fb010e9" (UID: "074ba060-2a84-479d-9d8b-92987fb010e9"). InnerVolumeSpecName "kube-api-access-gzrqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:15:14.847050 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.847038    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-envoy" pod "e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" (UID: "e17e1eaf-fae0-49af-bfe9-8fe6b04bc163"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:15:14.847249 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.847141    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-kube-api-access-wchpg" pod "e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" (UID: "e17e1eaf-fae0-49af-bfe9-8fe6b04bc163"). InnerVolumeSpecName "kube-api-access-wchpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:15:14.847249 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.847174    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/074ba060-2a84-479d-9d8b-92987fb010e9-istio-podinfo" pod "074ba060-2a84-479d-9d8b-92987fb010e9" (UID: "074ba060-2a84-479d-9d8b-92987fb010e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:15:14.847524 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.847503    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074ba060-2a84-479d-9d8b-92987fb010e9-istio-token" pod "074ba060-2a84-479d-9d8b-92987fb010e9" (UID: "074ba060-2a84-479d-9d8b-92987fb010e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:15:14.849348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.849296    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-istio-envoy" pod "074ba060-2a84-479d-9d8b-92987fb010e9" (UID: "074ba060-2a84-479d-9d8b-92987fb010e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:15:14.946208 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.946168    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.946208 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.946210    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.946440 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.946223    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.946440 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.946237    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.946440 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.946251    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/074ba060-2a84-479d-9d8b-92987fb010e9-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.946440 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.946264    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/074ba060-2a84-479d-9d8b-92987fb010e9-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.946440 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.946278    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.946440 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.946292    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/074ba060-2a84-479d-9d8b-92987fb010e9-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.946440 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.946306    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/074ba060-2a84-479d-9d8b-92987fb010e9-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.946440 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.946334    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wchpg\" (UniqueName: \"kubernetes.io/projected/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-kube-api-access-wchpg\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.946440 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.946351    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzrqq\" (UniqueName: \"kubernetes.io/projected/074ba060-2a84-479d-9d8b-92987fb010e9-kube-api-access-gzrqq\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:14.946440 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:14.946365    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:15.391771 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:15.391736    5619 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 01 06:15:15.391936 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:15.391798    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm" podUID="b44c28d4-b7ca-4350-b78a-53e4c9036124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 01 06:15:15.604133 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:15.604096    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"
Nov 01 06:15:15.604133 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:15.604116    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9" event={"ID":"e17e1eaf-fae0-49af-bfe9-8fe6b04bc163","Type":"ContainerDied","Data":"1fb5c71e5103f65b88a0881672768aedd552cd6f4100a299545ba0044629c1f2"}
Nov 01 06:15:15.604696 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:15.604193    5619 scope.go:122] "RemoveContainer" containerID="5aeb4125a45b0f389c81cc59878c22a41cef1e64aab13fa696d7240abe4802bd"
Nov 01 06:15:15.607122 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:15.607096    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs" event={"ID":"074ba060-2a84-479d-9d8b-92987fb010e9","Type":"ContainerDied","Data":"8edb94752d02bbb836b19933af0d96f5291390ef4158ca1a0806b2495e513f8b"}
Nov 01 06:15:15.607537 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:15.607291    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"
Nov 01 06:15:15.608777 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:15.608758    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"
Nov 01 06:15:15.608884 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:15.608822    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7" event={"ID":"231e12b3-f2c4-4633-8a9b-4dcfc8c34206","Type":"ContainerDied","Data":"f79b04ac50504f852e4744fe02e9a47cf2babdf63719418e172d5809fc1f5414"}
Nov 01 06:15:15.616433 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:15.616417    5619 scope.go:122] "RemoveContainer" containerID="bb33628316f779c952e16a48ffe64795df08196c9b221cad5055fcc016b56ec7"
Nov 01 06:15:15.625480 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:15.625463    5619 scope.go:122] "RemoveContainer" containerID="6e307a6f756f4ca43804d8d82e122eca5705ab8619e430f43d4cac8d515e6331"
Nov 01 06:15:15.780013 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:15.779973    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"]
Nov 01 06:15:15.829679 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:15.829647    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pskp9"]
Nov 01 06:15:15.892260 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:15.892175    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"]
Nov 01 06:15:15.896368 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:15.896344    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6xdr7"]
Nov 01 06:15:16.008028 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:16.007996    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"]
Nov 01 06:15:16.014031 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:16.013995    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qbzfs"]
Nov 01 06:15:16.396961 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:16.396811    5619 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 01 06:15:16.396961 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:16.396870    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm" podUID="b44c28d4-b7ca-4350-b78a-53e4c9036124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 01 06:15:16.963385 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:16.963349    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="074ba060-2a84-479d-9d8b-92987fb010e9" path="/var/lib/kubelet/pods/074ba060-2a84-479d-9d8b-92987fb010e9/volumes"
Nov 01 06:15:16.963962 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:16.963943    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="231e12b3-f2c4-4633-8a9b-4dcfc8c34206" path="/var/lib/kubelet/pods/231e12b3-f2c4-4633-8a9b-4dcfc8c34206/volumes"
Nov 01 06:15:16.964472 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:16.964455    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" path="/var/lib/kubelet/pods/e17e1eaf-fae0-49af-bfe9-8fe6b04bc163/volumes"
Nov 01 06:15:17.171742 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:17.171672    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:17.341203 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:17.341175    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh"]
Nov 01 06:15:17.397382 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:17.397341    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:17.397757 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:17.397734    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:17.398756 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:17.398732    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:15:17.445901 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:17.445868    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd"]
Nov 01 06:15:17.691662 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:17.691561    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:17.847110 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:17.847073    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:18.153598 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:18.153561    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:19.578573 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:19.578532    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:21.364542 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:21.364507    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:22.341690 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:22.341614    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh" podUID="0e66b197-c227-414d-8fa8-8f39a228fabd" containerName="payload-processing" containerID="cri-o://6010e2b0db05c73f2ab95f622bde176761efdae2a1701d36fde9256df5ba4cf2" gracePeriod=25
Nov 01 06:15:22.446616 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:22.446567    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd" podUID="bed3762c-2388-4654-9555-6c4cc3b817e3" containerName="payload-pre-processing" containerID="cri-o://e3c90461bc8a5f65c4c7073408080b809eae2a868ac01de2705f7cb32fe437e5" gracePeriod=25
Nov 01 06:15:22.757513 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:22.757425    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:25.882827 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:15:25.882785    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:15:25.917796 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:25.917766    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:26.089429 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:26.089387    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:26.229431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:26.229334    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:26.449497 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:26.449456    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd" podUID="bed3762c-2388-4654-9555-6c4cc3b817e3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:15:26.449694 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:26.449577    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh" podUID="0e66b197-c227-414d-8fa8-8f39a228fabd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:15:27.270852 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:27.270818    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:32.157578 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:32.157540    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:32.300474 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:32.300445    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:36.449879 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:36.449842    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd" podUID="bed3762c-2388-4654-9555-6c4cc3b817e3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:15:36.450260 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:36.449841    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh" podUID="0e66b197-c227-414d-8fa8-8f39a228fabd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:15:36.869567 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:36.869527    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:41.781632 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:41.781597    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:42.630890 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.630837    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh"
Nov 01 06:15:42.755708 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.755677    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e66b197-c227-414d-8fa8-8f39a228fabd-kube-api-access-vcds7\" (UniqueName: \"kubernetes.io/projected/0e66b197-c227-414d-8fa8-8f39a228fabd-kube-api-access-vcds7\") pod \"0e66b197-c227-414d-8fa8-8f39a228fabd\" (UID: \"0e66b197-c227-414d-8fa8-8f39a228fabd\") "
Nov 01 06:15:42.755978 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.755958    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e66b197-c227-414d-8fa8-8f39a228fabd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e66b197-c227-414d-8fa8-8f39a228fabd-plugins-config-volume\") pod \"0e66b197-c227-414d-8fa8-8f39a228fabd\" (UID: \"0e66b197-c227-414d-8fa8-8f39a228fabd\") "
Nov 01 06:15:42.756531 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.756492    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e66b197-c227-414d-8fa8-8f39a228fabd-plugins-config-volume" pod "0e66b197-c227-414d-8fa8-8f39a228fabd" (UID: "0e66b197-c227-414d-8fa8-8f39a228fabd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:15:42.756660 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.756631    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e66b197-c227-414d-8fa8-8f39a228fabd-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:42.766778 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.766751    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e66b197-c227-414d-8fa8-8f39a228fabd-kube-api-access-vcds7" pod "0e66b197-c227-414d-8fa8-8f39a228fabd" (UID: "0e66b197-c227-414d-8fa8-8f39a228fabd"). InnerVolumeSpecName "kube-api-access-vcds7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:15:42.769093 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.769066    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd"
Nov 01 06:15:42.796369 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.792708    5619 generic.go:356] "Generic (PLEG): container finished" podID="0e66b197-c227-414d-8fa8-8f39a228fabd" containerID="6010e2b0db05c73f2ab95f622bde176761efdae2a1701d36fde9256df5ba4cf2" exitCode=0
Nov 01 06:15:42.796369 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.792772    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh"
Nov 01 06:15:42.796369 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.792876    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh" event={"ID":"0e66b197-c227-414d-8fa8-8f39a228fabd","Type":"ContainerDied","Data":"6010e2b0db05c73f2ab95f622bde176761efdae2a1701d36fde9256df5ba4cf2"}
Nov 01 06:15:42.796369 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.792920    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh" event={"ID":"0e66b197-c227-414d-8fa8-8f39a228fabd","Type":"ContainerDied","Data":"15e516c399d59950145bad5b70391ff50efbf6cef96b8cbcd48caed831822ced"}
Nov 01 06:15:42.796369 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.792941    5619 scope.go:122] "RemoveContainer" containerID="6010e2b0db05c73f2ab95f622bde176761efdae2a1701d36fde9256df5ba4cf2"
Nov 01 06:15:42.796369 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.795444    5619 generic.go:356] "Generic (PLEG): container finished" podID="bed3762c-2388-4654-9555-6c4cc3b817e3" containerID="e3c90461bc8a5f65c4c7073408080b809eae2a868ac01de2705f7cb32fe437e5" exitCode=0
Nov 01 06:15:42.796369 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.795497    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd" event={"ID":"bed3762c-2388-4654-9555-6c4cc3b817e3","Type":"ContainerDied","Data":"e3c90461bc8a5f65c4c7073408080b809eae2a868ac01de2705f7cb32fe437e5"}
Nov 01 06:15:42.796369 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.795542    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd" event={"ID":"bed3762c-2388-4654-9555-6c4cc3b817e3","Type":"ContainerDied","Data":"b71341c1ef0dbd49217b9b67cae3493a3c57b66d318a3a5203a279dcf922ac06"}
Nov 01 06:15:42.796369 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.795619    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd"
Nov 01 06:15:42.807066 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.806985    5619 scope.go:122] "RemoveContainer" containerID="6010e2b0db05c73f2ab95f622bde176761efdae2a1701d36fde9256df5ba4cf2"
Nov 01 06:15:42.807471 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:15:42.807439    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6010e2b0db05c73f2ab95f622bde176761efdae2a1701d36fde9256df5ba4cf2\": container with ID starting with 6010e2b0db05c73f2ab95f622bde176761efdae2a1701d36fde9256df5ba4cf2 not found: ID does not exist" containerID="6010e2b0db05c73f2ab95f622bde176761efdae2a1701d36fde9256df5ba4cf2"
Nov 01 06:15:42.807581 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.807481    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6010e2b0db05c73f2ab95f622bde176761efdae2a1701d36fde9256df5ba4cf2"} err="failed to get container status \"6010e2b0db05c73f2ab95f622bde176761efdae2a1701d36fde9256df5ba4cf2\": rpc error: code = NotFound desc = could not find container \"6010e2b0db05c73f2ab95f622bde176761efdae2a1701d36fde9256df5ba4cf2\": container with ID starting with 6010e2b0db05c73f2ab95f622bde176761efdae2a1701d36fde9256df5ba4cf2 not found: ID does not exist"
Nov 01 06:15:42.807581 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.807505    5619 scope.go:122] "RemoveContainer" containerID="e3c90461bc8a5f65c4c7073408080b809eae2a868ac01de2705f7cb32fe437e5"
Nov 01 06:15:42.819446 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.819422    5619 scope.go:122] "RemoveContainer" containerID="e3c90461bc8a5f65c4c7073408080b809eae2a868ac01de2705f7cb32fe437e5"
Nov 01 06:15:42.819740 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:15:42.819716    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c90461bc8a5f65c4c7073408080b809eae2a868ac01de2705f7cb32fe437e5\": container with ID starting with e3c90461bc8a5f65c4c7073408080b809eae2a868ac01de2705f7cb32fe437e5 not found: ID does not exist" containerID="e3c90461bc8a5f65c4c7073408080b809eae2a868ac01de2705f7cb32fe437e5"
Nov 01 06:15:42.819830 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.819743    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c90461bc8a5f65c4c7073408080b809eae2a868ac01de2705f7cb32fe437e5"} err="failed to get container status \"e3c90461bc8a5f65c4c7073408080b809eae2a868ac01de2705f7cb32fe437e5\": rpc error: code = NotFound desc = could not find container \"e3c90461bc8a5f65c4c7073408080b809eae2a868ac01de2705f7cb32fe437e5\": container with ID starting with e3c90461bc8a5f65c4c7073408080b809eae2a868ac01de2705f7cb32fe437e5 not found: ID does not exist"
Nov 01 06:15:42.857960 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.857888    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bed3762c-2388-4654-9555-6c4cc3b817e3-kube-api-access-t7f6j\" (UniqueName: \"kubernetes.io/projected/bed3762c-2388-4654-9555-6c4cc3b817e3-kube-api-access-t7f6j\") pod \"bed3762c-2388-4654-9555-6c4cc3b817e3\" (UID: \"bed3762c-2388-4654-9555-6c4cc3b817e3\") "
Nov 01 06:15:42.858087 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.858002    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bed3762c-2388-4654-9555-6c4cc3b817e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bed3762c-2388-4654-9555-6c4cc3b817e3-plugins-config-volume\") pod \"bed3762c-2388-4654-9555-6c4cc3b817e3\" (UID: \"bed3762c-2388-4654-9555-6c4cc3b817e3\") "
Nov 01 06:15:42.858481 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.858453    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed3762c-2388-4654-9555-6c4cc3b817e3-plugins-config-volume" pod "bed3762c-2388-4654-9555-6c4cc3b817e3" (UID: "bed3762c-2388-4654-9555-6c4cc3b817e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:15:42.858590 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.858543    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bed3762c-2388-4654-9555-6c4cc3b817e3-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:42.858590 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.858567    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcds7\" (UniqueName: \"kubernetes.io/projected/0e66b197-c227-414d-8fa8-8f39a228fabd-kube-api-access-vcds7\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:42.860506 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.860486    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed3762c-2388-4654-9555-6c4cc3b817e3-kube-api-access-t7f6j" pod "bed3762c-2388-4654-9555-6c4cc3b817e3" (UID: "bed3762c-2388-4654-9555-6c4cc3b817e3"). InnerVolumeSpecName "kube-api-access-t7f6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:15:42.959709 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:42.959670    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7f6j\" (UniqueName: \"kubernetes.io/projected/bed3762c-2388-4654-9555-6c4cc3b817e3-kube-api-access-t7f6j\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:15:43.057123 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:43.057082    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh"]
Nov 01 06:15:43.062137 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:43.062106    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3578ce-79bc97d4cc-d26nh"]
Nov 01 06:15:43.217832 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:43.217755    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd"]
Nov 01 06:15:43.219954 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:43.219933    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3578ce-58f5455fc9-dmkdd"]
Nov 01 06:15:44.961716 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:44.961685    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e66b197-c227-414d-8fa8-8f39a228fabd" path="/var/lib/kubelet/pods/0e66b197-c227-414d-8fa8-8f39a228fabd/volumes"
Nov 01 06:15:44.962190 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:44.962146    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bed3762c-2388-4654-9555-6c4cc3b817e3" path="/var/lib/kubelet/pods/bed3762c-2388-4654-9555-6c4cc3b817e3/volumes"
Nov 01 06:15:46.557438 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:46.557401    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:51.361038 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:51.361001    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:55.882118 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:15:55.882064    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:15:56.090523 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:56.090491    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:56.380281 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:56.380246    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:57.022597 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:57.022564    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:57.169145 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:57.169110    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:15:57.307679 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:15:57.307584    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:00.969400 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:00.969369    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:01.627565 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:01.627532    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:01.823748 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:01.823711    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:04.846915 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:04.846860    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:05.023390 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:05.023347    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:06.213491 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:06.213449    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:09.155860 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:09.155825    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"]
Nov 01 06:16:09.156402 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:09.156104    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk" podUID="23561314-57a9-4def-a052-32c23ccdd1f0" containerName="istio-proxy" containerID="cri-o://6c345b575f576b09ada6941038a28c2158ba8b70c4778ec7312bdd361c10b233" gracePeriod=30
Nov 01 06:16:09.206725 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:09.206688    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"]
Nov 01 06:16:09.206892 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:09.206876    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5" podUID="39efefc9-9afa-4664-91f3-a86554a6e761" containerName="istio-proxy" containerID="cri-o://90032855d51c0ebf15285bc95e0d3c4c0df337b05d80d6e1aea976bdf900a221" gracePeriod=30
Nov 01 06:16:11.193060 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:11.192964    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:11.449051 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:11.448966    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:16:11.449051 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:11.449019    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk" podUID="23561314-57a9-4def-a052-32c23ccdd1f0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:16:14.487617 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.487592    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:16:14.512115 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.512095    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:16:14.660547 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.660490    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-credential-socket\") pod \"23561314-57a9-4def-a052-32c23ccdd1f0\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") "
Nov 01 06:16:14.660764 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.660578    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39efefc9-9afa-4664-91f3-a86554a6e761-istio-token\" (UniqueName: \"kubernetes.io/projected/39efefc9-9afa-4664-91f3-a86554a6e761-istio-token\") pod \"39efefc9-9afa-4664-91f3-a86554a6e761\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") "
Nov 01 06:16:14.660764 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.660619    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23561314-57a9-4def-a052-32c23ccdd1f0-kube-api-access-56spz\" (UniqueName: \"kubernetes.io/projected/23561314-57a9-4def-a052-32c23ccdd1f0-kube-api-access-56spz\") pod \"23561314-57a9-4def-a052-32c23ccdd1f0\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") "
Nov 01 06:16:14.660764 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.660671    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-istio-data\") pod \"23561314-57a9-4def-a052-32c23ccdd1f0\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") "
Nov 01 06:16:14.660764 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.660742    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39efefc9-9afa-4664-91f3-a86554a6e761-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39efefc9-9afa-4664-91f3-a86554a6e761-istio-podinfo\") pod \"39efefc9-9afa-4664-91f3-a86554a6e761\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") "
Nov 01 06:16:14.661019 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.660787    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-istio-data\") pod \"39efefc9-9afa-4664-91f3-a86554a6e761\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") "
Nov 01 06:16:14.661019 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.660815    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-istio-envoy\") pod \"39efefc9-9afa-4664-91f3-a86554a6e761\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") "
Nov 01 06:16:14.661019 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.660842    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-workload-socket\") pod \"39efefc9-9afa-4664-91f3-a86554a6e761\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") "
Nov 01 06:16:14.661019 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.660873    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39efefc9-9afa-4664-91f3-a86554a6e761-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39efefc9-9afa-4664-91f3-a86554a6e761-istiod-ca-cert\") pod \"39efefc9-9afa-4664-91f3-a86554a6e761\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") "
Nov 01 06:16:14.661019 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.660919    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23561314-57a9-4def-a052-32c23ccdd1f0-istio-token\" (UniqueName: \"kubernetes.io/projected/23561314-57a9-4def-a052-32c23ccdd1f0-istio-token\") pod \"23561314-57a9-4def-a052-32c23ccdd1f0\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") "
Nov 01 06:16:14.661019 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.660980    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-credential-socket\") pod \"39efefc9-9afa-4664-91f3-a86554a6e761\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") "
Nov 01 06:16:14.661289 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.661035    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-workload-certs\") pod \"39efefc9-9afa-4664-91f3-a86554a6e761\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") "
Nov 01 06:16:14.661289 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.661136    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-workload-certs\") pod \"23561314-57a9-4def-a052-32c23ccdd1f0\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") "
Nov 01 06:16:14.661289 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.661161    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-istio-envoy\") pod \"23561314-57a9-4def-a052-32c23ccdd1f0\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") "
Nov 01 06:16:14.661289 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.661189    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39efefc9-9afa-4664-91f3-a86554a6e761-kube-api-access-cwdjq\" (UniqueName: \"kubernetes.io/projected/39efefc9-9afa-4664-91f3-a86554a6e761-kube-api-access-cwdjq\") pod \"39efefc9-9afa-4664-91f3-a86554a6e761\" (UID: \"39efefc9-9afa-4664-91f3-a86554a6e761\") "
Nov 01 06:16:14.661289 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.661250    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-workload-socket\") pod \"23561314-57a9-4def-a052-32c23ccdd1f0\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") "
Nov 01 06:16:14.661550 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.661302    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23561314-57a9-4def-a052-32c23ccdd1f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23561314-57a9-4def-a052-32c23ccdd1f0-istio-podinfo\") pod \"23561314-57a9-4def-a052-32c23ccdd1f0\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") "
Nov 01 06:16:14.661550 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.661348    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23561314-57a9-4def-a052-32c23ccdd1f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23561314-57a9-4def-a052-32c23ccdd1f0-istiod-ca-cert\") pod \"23561314-57a9-4def-a052-32c23ccdd1f0\" (UID: \"23561314-57a9-4def-a052-32c23ccdd1f0\") "
Nov 01 06:16:14.661550 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.661476    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-istio-data" pod "23561314-57a9-4def-a052-32c23ccdd1f0" (UID: "23561314-57a9-4def-a052-32c23ccdd1f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:14.662312 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.661835    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23561314-57a9-4def-a052-32c23ccdd1f0-istiod-ca-cert" pod "23561314-57a9-4def-a052-32c23ccdd1f0" (UID: "23561314-57a9-4def-a052-32c23ccdd1f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:16:14.662312 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.662164    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-workload-socket" pod "39efefc9-9afa-4664-91f3-a86554a6e761" (UID: "39efefc9-9afa-4664-91f3-a86554a6e761"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:14.662610 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.662359    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-credential-socket" pod "23561314-57a9-4def-a052-32c23ccdd1f0" (UID: "23561314-57a9-4def-a052-32c23ccdd1f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:14.662610 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.662520    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-istio-data" pod "39efefc9-9afa-4664-91f3-a86554a6e761" (UID: "39efefc9-9afa-4664-91f3-a86554a6e761"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:14.663387 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.663361    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39efefc9-9afa-4664-91f3-a86554a6e761-istio-token" pod "39efefc9-9afa-4664-91f3-a86554a6e761" (UID: "39efefc9-9afa-4664-91f3-a86554a6e761"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:16:14.663604 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.663581    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-workload-certs" pod "39efefc9-9afa-4664-91f3-a86554a6e761" (UID: "39efefc9-9afa-4664-91f3-a86554a6e761"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:14.663735 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.663716    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-credential-socket" pod "39efefc9-9afa-4664-91f3-a86554a6e761" (UID: "39efefc9-9afa-4664-91f3-a86554a6e761"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:14.663901 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.663874    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-workload-certs" pod "23561314-57a9-4def-a052-32c23ccdd1f0" (UID: "23561314-57a9-4def-a052-32c23ccdd1f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:14.664083 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.664047    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-workload-socket" pod "23561314-57a9-4def-a052-32c23ccdd1f0" (UID: "23561314-57a9-4def-a052-32c23ccdd1f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:14.664652 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.664628    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-istio-envoy" pod "39efefc9-9afa-4664-91f3-a86554a6e761" (UID: "39efefc9-9afa-4664-91f3-a86554a6e761"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:14.665874 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.665850    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39efefc9-9afa-4664-91f3-a86554a6e761-kube-api-access-cwdjq" pod "39efefc9-9afa-4664-91f3-a86554a6e761" (UID: "39efefc9-9afa-4664-91f3-a86554a6e761"). InnerVolumeSpecName "kube-api-access-cwdjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:16:14.666156 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.666133    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39efefc9-9afa-4664-91f3-a86554a6e761-istio-podinfo" pod "39efefc9-9afa-4664-91f3-a86554a6e761" (UID: "39efefc9-9afa-4664-91f3-a86554a6e761"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:16:14.666156 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.666139    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23561314-57a9-4def-a052-32c23ccdd1f0-istio-token" pod "23561314-57a9-4def-a052-32c23ccdd1f0" (UID: "23561314-57a9-4def-a052-32c23ccdd1f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:16:14.666306 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.666282    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39efefc9-9afa-4664-91f3-a86554a6e761-istiod-ca-cert" pod "39efefc9-9afa-4664-91f3-a86554a6e761" (UID: "39efefc9-9afa-4664-91f3-a86554a6e761"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:16:14.666578 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.666560    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:14.666659 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.666583    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:14.666659 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.666597    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:14.666659 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.666612    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39efefc9-9afa-4664-91f3-a86554a6e761-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:14.666659 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.666627    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23561314-57a9-4def-a052-32c23ccdd1f0-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:14.666659 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.666641    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:14.666659 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.666654    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39efefc9-9afa-4664-91f3-a86554a6e761-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:14.666910 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.666668    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:14.666910 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.666682    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwdjq\" (UniqueName: \"kubernetes.io/projected/39efefc9-9afa-4664-91f3-a86554a6e761-kube-api-access-cwdjq\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:14.666910 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.666699    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:14.666910 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.666715    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23561314-57a9-4def-a052-32c23ccdd1f0-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:14.666910 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.666722    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-istio-envoy" pod "23561314-57a9-4def-a052-32c23ccdd1f0" (UID: "23561314-57a9-4def-a052-32c23ccdd1f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:16:14.666910 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.666729    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:14.666910 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.666755    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23561314-57a9-4def-a052-32c23ccdd1f0-istio-podinfo" pod "23561314-57a9-4def-a052-32c23ccdd1f0" (UID: "23561314-57a9-4def-a052-32c23ccdd1f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:16:14.666910 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.666769    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39efefc9-9afa-4664-91f3-a86554a6e761-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:14.666910 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.666788    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:14.666910 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.666806    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39efefc9-9afa-4664-91f3-a86554a6e761-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:14.667531 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.667505    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23561314-57a9-4def-a052-32c23ccdd1f0-kube-api-access-56spz" pod "23561314-57a9-4def-a052-32c23ccdd1f0" (UID: "23561314-57a9-4def-a052-32c23ccdd1f0"). InnerVolumeSpecName "kube-api-access-56spz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:16:14.767966 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.767887    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23561314-57a9-4def-a052-32c23ccdd1f0-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:14.767966 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.767917    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23561314-57a9-4def-a052-32c23ccdd1f0-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:14.767966 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.767927    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56spz\" (UniqueName: \"kubernetes.io/projected/23561314-57a9-4def-a052-32c23ccdd1f0-kube-api-access-56spz\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:14.994487 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.994455    5619 generic.go:356] "Generic (PLEG): container finished" podID="23561314-57a9-4def-a052-32c23ccdd1f0" containerID="6c345b575f576b09ada6941038a28c2158ba8b70c4778ec7312bdd361c10b233" exitCode=0
Nov 01 06:16:14.994641 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.994526    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"
Nov 01 06:16:14.994641 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.994519    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk" event={"ID":"23561314-57a9-4def-a052-32c23ccdd1f0","Type":"ContainerDied","Data":"6c345b575f576b09ada6941038a28c2158ba8b70c4778ec7312bdd361c10b233"}
Nov 01 06:16:14.994758 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.994708    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk" event={"ID":"23561314-57a9-4def-a052-32c23ccdd1f0","Type":"ContainerDied","Data":"b9609a754fa664849bfaf9f2f0d073a4ee50f092ca8bc16acbd6016108b494bd"}
Nov 01 06:16:14.994758 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.994744    5619 scope.go:122] "RemoveContainer" containerID="6c345b575f576b09ada6941038a28c2158ba8b70c4778ec7312bdd361c10b233"
Nov 01 06:16:14.996284 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.996258    5619 generic.go:356] "Generic (PLEG): container finished" podID="39efefc9-9afa-4664-91f3-a86554a6e761" containerID="90032855d51c0ebf15285bc95e0d3c4c0df337b05d80d6e1aea976bdf900a221" exitCode=0
Nov 01 06:16:14.996451 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.996350    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5" event={"ID":"39efefc9-9afa-4664-91f3-a86554a6e761","Type":"ContainerDied","Data":"90032855d51c0ebf15285bc95e0d3c4c0df337b05d80d6e1aea976bdf900a221"}
Nov 01 06:16:14.996451 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.996387    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5" event={"ID":"39efefc9-9afa-4664-91f3-a86554a6e761","Type":"ContainerDied","Data":"fd5c479dfc931d29c9a79f58d99d3a73449df2c6a222241cd70e4556ffb23693"}
Nov 01 06:16:14.996580 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:14.996462    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"
Nov 01 06:16:15.005507 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:15.005481    5619 scope.go:122] "RemoveContainer" containerID="6c345b575f576b09ada6941038a28c2158ba8b70c4778ec7312bdd361c10b233"
Nov 01 06:16:15.005895 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:16:15.005857    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c345b575f576b09ada6941038a28c2158ba8b70c4778ec7312bdd361c10b233\": container with ID starting with 6c345b575f576b09ada6941038a28c2158ba8b70c4778ec7312bdd361c10b233 not found: ID does not exist" containerID="6c345b575f576b09ada6941038a28c2158ba8b70c4778ec7312bdd361c10b233"
Nov 01 06:16:15.005991 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:15.005916    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c345b575f576b09ada6941038a28c2158ba8b70c4778ec7312bdd361c10b233"} err="failed to get container status \"6c345b575f576b09ada6941038a28c2158ba8b70c4778ec7312bdd361c10b233\": rpc error: code = NotFound desc = could not find container \"6c345b575f576b09ada6941038a28c2158ba8b70c4778ec7312bdd361c10b233\": container with ID starting with 6c345b575f576b09ada6941038a28c2158ba8b70c4778ec7312bdd361c10b233 not found: ID does not exist"
Nov 01 06:16:15.005991 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:15.005944    5619 scope.go:122] "RemoveContainer" containerID="90032855d51c0ebf15285bc95e0d3c4c0df337b05d80d6e1aea976bdf900a221"
Nov 01 06:16:15.014727 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:15.014705    5619 scope.go:122] "RemoveContainer" containerID="90032855d51c0ebf15285bc95e0d3c4c0df337b05d80d6e1aea976bdf900a221"
Nov 01 06:16:15.014984 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:16:15.014965    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90032855d51c0ebf15285bc95e0d3c4c0df337b05d80d6e1aea976bdf900a221\": container with ID starting with 90032855d51c0ebf15285bc95e0d3c4c0df337b05d80d6e1aea976bdf900a221 not found: ID does not exist" containerID="90032855d51c0ebf15285bc95e0d3c4c0df337b05d80d6e1aea976bdf900a221"
Nov 01 06:16:15.015043 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:15.014991    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90032855d51c0ebf15285bc95e0d3c4c0df337b05d80d6e1aea976bdf900a221"} err="failed to get container status \"90032855d51c0ebf15285bc95e0d3c4c0df337b05d80d6e1aea976bdf900a221\": rpc error: code = NotFound desc = could not find container \"90032855d51c0ebf15285bc95e0d3c4c0df337b05d80d6e1aea976bdf900a221\": container with ID starting with 90032855d51c0ebf15285bc95e0d3c4c0df337b05d80d6e1aea976bdf900a221 not found: ID does not exist"
Nov 01 06:16:15.163347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:15.163303    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"]
Nov 01 06:16:15.214105 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:15.214071    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-dtpvk"]
Nov 01 06:16:15.280439 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:15.280406    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"]
Nov 01 06:16:15.284577 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:15.284548    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-fd7h5"]
Nov 01 06:16:16.059727 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:16.059462    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:16.964536 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:16.964501    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23561314-57a9-4def-a052-32c23ccdd1f0" path="/var/lib/kubelet/pods/23561314-57a9-4def-a052-32c23ccdd1f0/volumes"
Nov 01 06:16:16.965158 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:16.965132    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39efefc9-9afa-4664-91f3-a86554a6e761" path="/var/lib/kubelet/pods/39efefc9-9afa-4664-91f3-a86554a6e761/volumes"
Nov 01 06:16:20.769835 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:20.769792    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:21.719153 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:21.719119    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:25.554505 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:25.554459    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:25.883487 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:16:25.883447    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:16:26.531830 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:26.531796    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:30.276062 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:30.276025    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:35.164210 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:35.164169    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:39.950106 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:39.950063    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:44.609190 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:44.609148    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:44.801127 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:44.801089    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:44.942874 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:44.942777    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:45.735381 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:45.735344    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:49.564998 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:49.564960    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:54.363634 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:54.363594    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:55.881420 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:16:55.881373    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:16:58.409757 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:58.409726    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"]
Nov 01 06:16:58.410183 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:58.409926    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h" podUID="19041e40-ee3c-4e90-908e-e3072a30e985" containerName="maas-api" containerID="cri-o://e207e20a59f25d4388dad84246b6842470951b78881d5d9a3f6fa3cd56dbed4c" gracePeriod=30
Nov 01 06:16:58.510786 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:58.510749    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:16:58.691664 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:58.691639    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"
Nov 01 06:16:58.785741 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:58.785710    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19041e40-ee3c-4e90-908e-e3072a30e985-kube-api-access-cj7x2\" (UniqueName: \"kubernetes.io/projected/19041e40-ee3c-4e90-908e-e3072a30e985-kube-api-access-cj7x2\") pod \"19041e40-ee3c-4e90-908e-e3072a30e985\" (UID: \"19041e40-ee3c-4e90-908e-e3072a30e985\") "
Nov 01 06:16:58.786060 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:58.785799    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19041e40-ee3c-4e90-908e-e3072a30e985-metrics-certs\" (UniqueName: \"kubernetes.io/secret/19041e40-ee3c-4e90-908e-e3072a30e985-metrics-certs\") pod \"19041e40-ee3c-4e90-908e-e3072a30e985\" (UID: \"19041e40-ee3c-4e90-908e-e3072a30e985\") "
Nov 01 06:16:58.786060 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:58.785880    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19041e40-ee3c-4e90-908e-e3072a30e985-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19041e40-ee3c-4e90-908e-e3072a30e985-openshift-service-ca\") pod \"19041e40-ee3c-4e90-908e-e3072a30e985\" (UID: \"19041e40-ee3c-4e90-908e-e3072a30e985\") "
Nov 01 06:16:58.786204 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:58.786127    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19041e40-ee3c-4e90-908e-e3072a30e985-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19041e40-ee3c-4e90-908e-e3072a30e985-maas-api-tls\") pod \"19041e40-ee3c-4e90-908e-e3072a30e985\" (UID: \"19041e40-ee3c-4e90-908e-e3072a30e985\") "
Nov 01 06:16:58.786971 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:58.786916    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19041e40-ee3c-4e90-908e-e3072a30e985-openshift-service-ca" pod "19041e40-ee3c-4e90-908e-e3072a30e985" (UID: "19041e40-ee3c-4e90-908e-e3072a30e985"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:16:58.806414 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:58.794656    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19041e40-ee3c-4e90-908e-e3072a30e985-maas-api-tls" pod "19041e40-ee3c-4e90-908e-e3072a30e985" (UID: "19041e40-ee3c-4e90-908e-e3072a30e985"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:16:58.806414 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:58.795107    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19041e40-ee3c-4e90-908e-e3072a30e985-kube-api-access-cj7x2" pod "19041e40-ee3c-4e90-908e-e3072a30e985" (UID: "19041e40-ee3c-4e90-908e-e3072a30e985"). InnerVolumeSpecName "kube-api-access-cj7x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:16:58.806414 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:58.802867    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19041e40-ee3c-4e90-908e-e3072a30e985-metrics-certs" pod "19041e40-ee3c-4e90-908e-e3072a30e985" (UID: "19041e40-ee3c-4e90-908e-e3072a30e985"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:16:58.888406 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:58.888371    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj7x2\" (UniqueName: \"kubernetes.io/projected/19041e40-ee3c-4e90-908e-e3072a30e985-kube-api-access-cj7x2\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:58.888406 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:58.888405    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19041e40-ee3c-4e90-908e-e3072a30e985-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:58.888603 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:58.888419    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19041e40-ee3c-4e90-908e-e3072a30e985-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:58.888603 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:58.888433    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19041e40-ee3c-4e90-908e-e3072a30e985-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:16:59.265209 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:59.265122    5619 generic.go:356] "Generic (PLEG): container finished" podID="19041e40-ee3c-4e90-908e-e3072a30e985" containerID="e207e20a59f25d4388dad84246b6842470951b78881d5d9a3f6fa3cd56dbed4c" exitCode=0
Nov 01 06:16:59.265209 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:59.265185    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h" event={"ID":"19041e40-ee3c-4e90-908e-e3072a30e985","Type":"ContainerDied","Data":"e207e20a59f25d4388dad84246b6842470951b78881d5d9a3f6fa3cd56dbed4c"}
Nov 01 06:16:59.265482 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:59.265201    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"
Nov 01 06:16:59.265482 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:59.265271    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h" event={"ID":"19041e40-ee3c-4e90-908e-e3072a30e985","Type":"ContainerDied","Data":"57eb3108f6bf597b568909b8592b7e118d6eb23e790454436b5bad02bb4c9abe"}
Nov 01 06:16:59.265482 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:59.265312    5619 scope.go:122] "RemoveContainer" containerID="e207e20a59f25d4388dad84246b6842470951b78881d5d9a3f6fa3cd56dbed4c"
Nov 01 06:16:59.276791 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:59.276769    5619 scope.go:122] "RemoveContainer" containerID="e207e20a59f25d4388dad84246b6842470951b78881d5d9a3f6fa3cd56dbed4c"
Nov 01 06:16:59.277092 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:16:59.277066    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e207e20a59f25d4388dad84246b6842470951b78881d5d9a3f6fa3cd56dbed4c\": container with ID starting with e207e20a59f25d4388dad84246b6842470951b78881d5d9a3f6fa3cd56dbed4c not found: ID does not exist" containerID="e207e20a59f25d4388dad84246b6842470951b78881d5d9a3f6fa3cd56dbed4c"
Nov 01 06:16:59.277159 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:59.277105    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e207e20a59f25d4388dad84246b6842470951b78881d5d9a3f6fa3cd56dbed4c"} err="failed to get container status \"e207e20a59f25d4388dad84246b6842470951b78881d5d9a3f6fa3cd56dbed4c\": rpc error: code = NotFound desc = could not find container \"e207e20a59f25d4388dad84246b6842470951b78881d5d9a3f6fa3cd56dbed4c\": container with ID starting with e207e20a59f25d4388dad84246b6842470951b78881d5d9a3f6fa3cd56dbed4c not found: ID does not exist"
Nov 01 06:16:59.440534 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:59.440496    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"]
Nov 01 06:16:59.491036 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:16:59.491008    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3578ce-6755fd9b5-bk46h"]
Nov 01 06:17:00.252485 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:00.252449    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:00.439845 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:00.439810    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:00.704297 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:00.704259    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:00.713011 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:00.712979    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"]
Nov 01 06:17:00.713308 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:00.713271    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx" podUID="7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" containerName="istio-proxy" containerID="cri-o://3a0eaae25e8f8e239622ae9c98075bf543d3f9089a3e1e9abdd8648e5fa182cf" gracePeriod=30
Nov 01 06:17:00.962432 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:00.962351    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19041e40-ee3c-4e90-908e-e3072a30e985" path="/var/lib/kubelet/pods/19041e40-ee3c-4e90-908e-e3072a30e985/volumes"
Nov 01 06:17:04.653033 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:04.652990    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:04.808120 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:04.808078    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:05.355404 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:05.355363    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:05.988720 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:05.988649    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:17:06.090979 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.090947    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istiod-ca-cert\") pod \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") "
Nov 01 06:17:06.091156 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.091006    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-workload-socket\") pod \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") "
Nov 01 06:17:06.091156 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.091071    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-envoy\") pod \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") "
Nov 01 06:17:06.091271 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.091173    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-kube-api-access-7vwbm\" (UniqueName: \"kubernetes.io/projected/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-kube-api-access-7vwbm\") pod \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") "
Nov 01 06:17:06.091271 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.091222    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-workload-certs\") pod \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") "
Nov 01 06:17:06.091413 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.091280    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-data\") pod \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") "
Nov 01 06:17:06.091413 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.091277    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-workload-socket" pod "7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" (UID: "7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:17:06.091413 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.091349    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-podinfo\") pod \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") "
Nov 01 06:17:06.091413 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.091379    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istiod-ca-cert" pod "7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" (UID: "7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:17:06.091613 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.091424    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-token\" (UniqueName: \"kubernetes.io/projected/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-token\") pod \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") "
Nov 01 06:17:06.091613 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.091548    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-workload-certs" pod "7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" (UID: "7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:17:06.091727 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.091600    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-data" pod "7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" (UID: "7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:17:06.091829 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.091811    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-credential-socket\") pod \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\" (UID: \"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea\") "
Nov 01 06:17:06.092115 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.092089    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-credential-socket" pod "7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" (UID: "7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:17:06.092403 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.092380    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:06.092453 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.092411    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:06.092453 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.092431    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:06.092453 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.092446    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:06.092556 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.092461    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:06.093509 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.093491    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-kube-api-access-7vwbm" pod "7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" (UID: "7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea"). InnerVolumeSpecName "kube-api-access-7vwbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:17:06.093711 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.093685    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-podinfo" pod "7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" (UID: "7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:17:06.093770 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.093702    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-envoy" pod "7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" (UID: "7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:17:06.093832 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.093809    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-token" pod "7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" (UID: "7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:17:06.193788 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.193758    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:06.193788 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.193787    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vwbm\" (UniqueName: \"kubernetes.io/projected/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-kube-api-access-7vwbm\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:06.193974 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.193800    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:06.193974 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.193809    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:06.307886 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.307806    5619 generic.go:356] "Generic (PLEG): container finished" podID="7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" containerID="3a0eaae25e8f8e239622ae9c98075bf543d3f9089a3e1e9abdd8648e5fa182cf" exitCode=0
Nov 01 06:17:06.307886 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.307865    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"
Nov 01 06:17:06.308079 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.307894    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx" event={"ID":"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea","Type":"ContainerDied","Data":"3a0eaae25e8f8e239622ae9c98075bf543d3f9089a3e1e9abdd8648e5fa182cf"}
Nov 01 06:17:06.308079 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.307982    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx" event={"ID":"7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea","Type":"ContainerDied","Data":"e838ed9b57ade69fc88e68599424fe8afadf723e66d9a3a67658e1ca6d35809f"}
Nov 01 06:17:06.308079 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.308013    5619 scope.go:122] "RemoveContainer" containerID="3a0eaae25e8f8e239622ae9c98075bf543d3f9089a3e1e9abdd8648e5fa182cf"
Nov 01 06:17:06.318119 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.318101    5619 scope.go:122] "RemoveContainer" containerID="d2bd719db01ac1dcbd2a8a8419906442d87c59664861e21ab80063beae02eac5"
Nov 01 06:17:06.327115 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.327091    5619 scope.go:122] "RemoveContainer" containerID="3a0eaae25e8f8e239622ae9c98075bf543d3f9089a3e1e9abdd8648e5fa182cf"
Nov 01 06:17:06.327419 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:17:06.327398    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a0eaae25e8f8e239622ae9c98075bf543d3f9089a3e1e9abdd8648e5fa182cf\": container with ID starting with 3a0eaae25e8f8e239622ae9c98075bf543d3f9089a3e1e9abdd8648e5fa182cf not found: ID does not exist" containerID="3a0eaae25e8f8e239622ae9c98075bf543d3f9089a3e1e9abdd8648e5fa182cf"
Nov 01 06:17:06.327487 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.327428    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a0eaae25e8f8e239622ae9c98075bf543d3f9089a3e1e9abdd8648e5fa182cf"} err="failed to get container status \"3a0eaae25e8f8e239622ae9c98075bf543d3f9089a3e1e9abdd8648e5fa182cf\": rpc error: code = NotFound desc = could not find container \"3a0eaae25e8f8e239622ae9c98075bf543d3f9089a3e1e9abdd8648e5fa182cf\": container with ID starting with 3a0eaae25e8f8e239622ae9c98075bf543d3f9089a3e1e9abdd8648e5fa182cf not found: ID does not exist"
Nov 01 06:17:06.327487 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.327447    5619 scope.go:122] "RemoveContainer" containerID="d2bd719db01ac1dcbd2a8a8419906442d87c59664861e21ab80063beae02eac5"
Nov 01 06:17:06.327704 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:17:06.327680    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2bd719db01ac1dcbd2a8a8419906442d87c59664861e21ab80063beae02eac5\": container with ID starting with d2bd719db01ac1dcbd2a8a8419906442d87c59664861e21ab80063beae02eac5 not found: ID does not exist" containerID="d2bd719db01ac1dcbd2a8a8419906442d87c59664861e21ab80063beae02eac5"
Nov 01 06:17:06.327738 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.327711    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2bd719db01ac1dcbd2a8a8419906442d87c59664861e21ab80063beae02eac5"} err="failed to get container status \"d2bd719db01ac1dcbd2a8a8419906442d87c59664861e21ab80063beae02eac5\": rpc error: code = NotFound desc = could not find container \"d2bd719db01ac1dcbd2a8a8419906442d87c59664861e21ab80063beae02eac5\": container with ID starting with d2bd719db01ac1dcbd2a8a8419906442d87c59664861e21ab80063beae02eac5 not found: ID does not exist"
Nov 01 06:17:06.477381 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.477340    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"]
Nov 01 06:17:06.481364 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.481339    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3578ce-openshift-default-b658d99d9-c9kwx"]
Nov 01 06:17:06.964911 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:06.964881    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" path="/var/lib/kubelet/pods/7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea/volumes"
Nov 01 06:17:10.072450 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:10.072409    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:14.851483 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:14.851444    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:16.210013 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:16.209967    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:19.556722 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:19.556677    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:22.065794 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:22.065756    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:22.627817 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:22.627778    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:23.140490 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:23.140452    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:23.332469 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:23.332414    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:24.045163 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.045129    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"]
Nov 01 06:17:24.045388 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.045356    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9" podUID="3eb9ca80-f436-41b4-8354-ba7498f47580" containerName="discovery" containerID="cri-o://01cad546c840c7fd3f0f85d7d8739d4e5501c2aa8041bf7e570079458feede78" gracePeriod=30
Nov 01 06:17:24.233137 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.232794    5619 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-c4ks9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:8080/ready\": dial tcp 10.133.0.81:8080: connect: connection refused" start-of-body=
Nov 01 06:17:24.233137 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.232873    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9" podUID="3eb9ca80-f436-41b4-8354-ba7498f47580" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.81:8080/ready\": dial tcp 10.133.0.81:8080: connect: connection refused"
Nov 01 06:17:24.378080 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.377765    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:17:24.419217 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.419179    5619 generic.go:356] "Generic (PLEG): container finished" podID="3eb9ca80-f436-41b4-8354-ba7498f47580" containerID="01cad546c840c7fd3f0f85d7d8739d4e5501c2aa8041bf7e570079458feede78" exitCode=0
Nov 01 06:17:24.419444 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.419270    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9" event={"ID":"3eb9ca80-f436-41b4-8354-ba7498f47580","Type":"ContainerDied","Data":"01cad546c840c7fd3f0f85d7d8739d4e5501c2aa8041bf7e570079458feede78"}
Nov 01 06:17:24.419444 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.419290    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"
Nov 01 06:17:24.419444 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.419360    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9" event={"ID":"3eb9ca80-f436-41b4-8354-ba7498f47580","Type":"ContainerDied","Data":"6fce4672e2f4193d7f23938c357ce3dfd655ed7a9f16637904268194adf64d50"}
Nov 01 06:17:24.419444 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.419385    5619 scope.go:122] "RemoveContainer" containerID="01cad546c840c7fd3f0f85d7d8739d4e5501c2aa8041bf7e570079458feede78"
Nov 01 06:17:24.431644 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.431621    5619 scope.go:122] "RemoveContainer" containerID="01cad546c840c7fd3f0f85d7d8739d4e5501c2aa8041bf7e570079458feede78"
Nov 01 06:17:24.431944 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:17:24.431925    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01cad546c840c7fd3f0f85d7d8739d4e5501c2aa8041bf7e570079458feede78\": container with ID starting with 01cad546c840c7fd3f0f85d7d8739d4e5501c2aa8041bf7e570079458feede78 not found: ID does not exist" containerID="01cad546c840c7fd3f0f85d7d8739d4e5501c2aa8041bf7e570079458feede78"
Nov 01 06:17:24.432033 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.431955    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01cad546c840c7fd3f0f85d7d8739d4e5501c2aa8041bf7e570079458feede78"} err="failed to get container status \"01cad546c840c7fd3f0f85d7d8739d4e5501c2aa8041bf7e570079458feede78\": rpc error: code = NotFound desc = could not find container \"01cad546c840c7fd3f0f85d7d8739d4e5501c2aa8041bf7e570079458feede78\": container with ID starting with 01cad546c840c7fd3f0f85d7d8739d4e5501c2aa8041bf7e570079458feede78 not found: ID does not exist"
Nov 01 06:17:24.513287 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.513249    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb9ca80-f436-41b4-8354-ba7498f47580-cacerts\" (UniqueName: \"kubernetes.io/secret/3eb9ca80-f436-41b4-8354-ba7498f47580-cacerts\") pod \"3eb9ca80-f436-41b4-8354-ba7498f47580\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") "
Nov 01 06:17:24.513478 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.513345    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-token\" (UniqueName: \"kubernetes.io/projected/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-token\") pod \"3eb9ca80-f436-41b4-8354-ba7498f47580\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") "
Nov 01 06:17:24.513478 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.513431    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb9ca80-f436-41b4-8354-ba7498f47580-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb9ca80-f436-41b4-8354-ba7498f47580-local-certs\") pod \"3eb9ca80-f436-41b4-8354-ba7498f47580\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") "
Nov 01 06:17:24.513578 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.513485    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-kubeconfig\") pod \"3eb9ca80-f436-41b4-8354-ba7498f47580\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") "
Nov 01 06:17:24.513578 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.513517    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-csr-dns-cert\") pod \"3eb9ca80-f436-41b4-8354-ba7498f47580\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") "
Nov 01 06:17:24.514927 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.514127    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb9ca80-f436-41b4-8354-ba7498f47580-kube-api-access-pdt6c\" (UniqueName: \"kubernetes.io/projected/3eb9ca80-f436-41b4-8354-ba7498f47580-kube-api-access-pdt6c\") pod \"3eb9ca80-f436-41b4-8354-ba7498f47580\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") "
Nov 01 06:17:24.514927 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.514187    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-csr-ca-configmap\") pod \"3eb9ca80-f436-41b4-8354-ba7498f47580\" (UID: \"3eb9ca80-f436-41b4-8354-ba7498f47580\") "
Nov 01 06:17:24.515606 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.515564    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-csr-ca-configmap" pod "3eb9ca80-f436-41b4-8354-ba7498f47580" (UID: "3eb9ca80-f436-41b4-8354-ba7498f47580"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:17:24.516070 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.516038    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb9ca80-f436-41b4-8354-ba7498f47580-cacerts" pod "3eb9ca80-f436-41b4-8354-ba7498f47580" (UID: "3eb9ca80-f436-41b4-8354-ba7498f47580"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:17:24.516287 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.516253    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-csr-dns-cert" pod "3eb9ca80-f436-41b4-8354-ba7498f47580" (UID: "3eb9ca80-f436-41b4-8354-ba7498f47580"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:17:24.516486 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.516370    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-token" pod "3eb9ca80-f436-41b4-8354-ba7498f47580" (UID: "3eb9ca80-f436-41b4-8354-ba7498f47580"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:17:24.516486 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.516390    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-kubeconfig" pod "3eb9ca80-f436-41b4-8354-ba7498f47580" (UID: "3eb9ca80-f436-41b4-8354-ba7498f47580"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:17:24.516725 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.516710    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb9ca80-f436-41b4-8354-ba7498f47580-local-certs" pod "3eb9ca80-f436-41b4-8354-ba7498f47580" (UID: "3eb9ca80-f436-41b4-8354-ba7498f47580"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:17:24.517758 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.517733    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb9ca80-f436-41b4-8354-ba7498f47580-kube-api-access-pdt6c" pod "3eb9ca80-f436-41b4-8354-ba7498f47580" (UID: "3eb9ca80-f436-41b4-8354-ba7498f47580"). InnerVolumeSpecName "kube-api-access-pdt6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:17:24.535591 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.535556    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:24.619421 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.617057    5619 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb9ca80-f436-41b4-8354-ba7498f47580-local-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:24.619421 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.617096    5619 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-kubeconfig\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:24.619421 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.617117    5619 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-csr-dns-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:24.619421 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.617133    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdt6c\" (UniqueName: \"kubernetes.io/projected/3eb9ca80-f436-41b4-8354-ba7498f47580-kube-api-access-pdt6c\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:24.619421 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.617149    5619 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-csr-ca-configmap\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:24.619421 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.617163    5619 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3eb9ca80-f436-41b4-8354-ba7498f47580-cacerts\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:24.619421 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.617177    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb9ca80-f436-41b4-8354-ba7498f47580-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:24.843499 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.843469    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"]
Nov 01 06:17:24.846863 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.846839    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c4ks9"]
Nov 01 06:17:24.961730 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:24.961659    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb9ca80-f436-41b4-8354-ba7498f47580" path="/var/lib/kubelet/pods/3eb9ca80-f436-41b4-8354-ba7498f47580/volumes"
Nov 01 06:17:25.880730 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:17:25.880691    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:17:26.881765 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:26.881729    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:28.368184 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:28.368138    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:29.382256 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:29.382219    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:34.091880 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:34.091841    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:38.115032 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.115000    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"]
Nov 01 06:17:38.115445 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.115179    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn" podUID="d21d21dc-79f2-40c1-b701-e75e87bd144c" containerName="maas-api" containerID="cri-o://cfd4f9b85f953e5b599a89f45c846b7fbe798187c2fe583967d66fbc4de3b609" gracePeriod=30
Nov 01 06:17:38.218813 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.218667    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:38.445518 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.445489    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:17:38.487476 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.487438    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d21d21dc-79f2-40c1-b701-e75e87bd144c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d21d21dc-79f2-40c1-b701-e75e87bd144c-maas-api-tls\") pod \"d21d21dc-79f2-40c1-b701-e75e87bd144c\" (UID: \"d21d21dc-79f2-40c1-b701-e75e87bd144c\") "
Nov 01 06:17:38.487664 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.487507    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d21d21dc-79f2-40c1-b701-e75e87bd144c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d21d21dc-79f2-40c1-b701-e75e87bd144c-openshift-service-ca\") pod \"d21d21dc-79f2-40c1-b701-e75e87bd144c\" (UID: \"d21d21dc-79f2-40c1-b701-e75e87bd144c\") "
Nov 01 06:17:38.487664 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.487529    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d21d21dc-79f2-40c1-b701-e75e87bd144c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d21d21dc-79f2-40c1-b701-e75e87bd144c-metrics-certs\") pod \"d21d21dc-79f2-40c1-b701-e75e87bd144c\" (UID: \"d21d21dc-79f2-40c1-b701-e75e87bd144c\") "
Nov 01 06:17:38.487664 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.487642    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d21d21dc-79f2-40c1-b701-e75e87bd144c-kube-api-access-6cmqp\" (UniqueName: \"kubernetes.io/projected/d21d21dc-79f2-40c1-b701-e75e87bd144c-kube-api-access-6cmqp\") pod \"d21d21dc-79f2-40c1-b701-e75e87bd144c\" (UID: \"d21d21dc-79f2-40c1-b701-e75e87bd144c\") "
Nov 01 06:17:38.488050 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.488021    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d21d21dc-79f2-40c1-b701-e75e87bd144c-openshift-service-ca" pod "d21d21dc-79f2-40c1-b701-e75e87bd144c" (UID: "d21d21dc-79f2-40c1-b701-e75e87bd144c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:17:38.489897 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.489867    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21d21dc-79f2-40c1-b701-e75e87bd144c-metrics-certs" pod "d21d21dc-79f2-40c1-b701-e75e87bd144c" (UID: "d21d21dc-79f2-40c1-b701-e75e87bd144c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:17:38.489991 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.489949    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21d21dc-79f2-40c1-b701-e75e87bd144c-kube-api-access-6cmqp" pod "d21d21dc-79f2-40c1-b701-e75e87bd144c" (UID: "d21d21dc-79f2-40c1-b701-e75e87bd144c"). InnerVolumeSpecName "kube-api-access-6cmqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:17:38.490094 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.490067    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21d21dc-79f2-40c1-b701-e75e87bd144c-maas-api-tls" pod "d21d21dc-79f2-40c1-b701-e75e87bd144c" (UID: "d21d21dc-79f2-40c1-b701-e75e87bd144c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:17:38.504627 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.504599    5619 generic.go:356] "Generic (PLEG): container finished" podID="d21d21dc-79f2-40c1-b701-e75e87bd144c" containerID="cfd4f9b85f953e5b599a89f45c846b7fbe798187c2fe583967d66fbc4de3b609" exitCode=0
Nov 01 06:17:38.504739 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.504658    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"
Nov 01 06:17:38.504739 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.504678    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn" event={"ID":"d21d21dc-79f2-40c1-b701-e75e87bd144c","Type":"ContainerDied","Data":"cfd4f9b85f953e5b599a89f45c846b7fbe798187c2fe583967d66fbc4de3b609"}
Nov 01 06:17:38.504848 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.504739    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn" event={"ID":"d21d21dc-79f2-40c1-b701-e75e87bd144c","Type":"ContainerDied","Data":"d626e477f8209c3302f2ff41054fe874b6ac8ae2d23c1e1605b73da5c6d35511"}
Nov 01 06:17:38.504848 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.504767    5619 scope.go:122] "RemoveContainer" containerID="cfd4f9b85f953e5b599a89f45c846b7fbe798187c2fe583967d66fbc4de3b609"
Nov 01 06:17:38.524265 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.524242    5619 scope.go:122] "RemoveContainer" containerID="cfd4f9b85f953e5b599a89f45c846b7fbe798187c2fe583967d66fbc4de3b609"
Nov 01 06:17:38.524594 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:17:38.524573    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfd4f9b85f953e5b599a89f45c846b7fbe798187c2fe583967d66fbc4de3b609\": container with ID starting with cfd4f9b85f953e5b599a89f45c846b7fbe798187c2fe583967d66fbc4de3b609 not found: ID does not exist" containerID="cfd4f9b85f953e5b599a89f45c846b7fbe798187c2fe583967d66fbc4de3b609"
Nov 01 06:17:38.524657 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.524603    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd4f9b85f953e5b599a89f45c846b7fbe798187c2fe583967d66fbc4de3b609"} err="failed to get container status \"cfd4f9b85f953e5b599a89f45c846b7fbe798187c2fe583967d66fbc4de3b609\": rpc error: code = NotFound desc = could not find container \"cfd4f9b85f953e5b599a89f45c846b7fbe798187c2fe583967d66fbc4de3b609\": container with ID starting with cfd4f9b85f953e5b599a89f45c846b7fbe798187c2fe583967d66fbc4de3b609 not found: ID does not exist"
Nov 01 06:17:38.590174 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.590055    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cmqp\" (UniqueName: \"kubernetes.io/projected/d21d21dc-79f2-40c1-b701-e75e87bd144c-kube-api-access-6cmqp\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:38.590174 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.590103    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d21d21dc-79f2-40c1-b701-e75e87bd144c-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:38.590174 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.590119    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d21d21dc-79f2-40c1-b701-e75e87bd144c-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:38.590174 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.590132    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d21d21dc-79f2-40c1-b701-e75e87bd144c-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:38.679685 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.679581    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"]
Nov 01 06:17:38.683135 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.683102    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec5596-65b6ff74bd-gfsvn"]
Nov 01 06:17:38.962852 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:38.962765    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d21d21dc-79f2-40c1-b701-e75e87bd144c" path="/var/lib/kubelet/pods/d21d21dc-79f2-40c1-b701-e75e87bd144c/volumes"
Nov 01 06:17:40.015349 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:40.015301    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:42.884204 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:42.884107    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:43.039964 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:43.039898    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:43.461188 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:43.461153    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:43.877860 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:43.877829    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:44.386964 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.386930    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"]
Nov 01 06:17:44.387655 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.387638    5619 cpu_manager.go:414] "containerMap: removing container" podUID="7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" containerName="istio-proxy"
Nov 01 06:17:44.387706 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.387657    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" containerName="istio-proxy"
Nov 01 06:17:44.387706 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.387672    5619 cpu_manager.go:414] "containerMap: removing container" podUID="231e12b3-f2c4-4633-8a9b-4dcfc8c34206" containerName="istio-proxy"
Nov 01 06:17:44.387706 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.387680    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231e12b3-f2c4-4633-8a9b-4dcfc8c34206" containerName="istio-proxy"
Nov 01 06:17:44.387706 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.387700    5619 cpu_manager.go:414] "containerMap: removing container" podUID="074ba060-2a84-479d-9d8b-92987fb010e9" containerName="istio-proxy"
Nov 01 06:17:44.387834 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.387708    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="074ba060-2a84-479d-9d8b-92987fb010e9" containerName="istio-proxy"
Nov 01 06:17:44.387834 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.387722    5619 cpu_manager.go:414] "containerMap: removing container" podUID="d21d21dc-79f2-40c1-b701-e75e87bd144c" containerName="maas-api"
Nov 01 06:17:44.387834 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.387730    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d21d21dc-79f2-40c1-b701-e75e87bd144c" containerName="maas-api"
Nov 01 06:17:44.387834 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.387751    5619 cpu_manager.go:414] "containerMap: removing container" podUID="23561314-57a9-4def-a052-32c23ccdd1f0" containerName="istio-proxy"
Nov 01 06:17:44.387834 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.387758    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23561314-57a9-4def-a052-32c23ccdd1f0" containerName="istio-proxy"
Nov 01 06:17:44.387834 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.387776    5619 cpu_manager.go:414] "containerMap: removing container" podUID="39efefc9-9afa-4664-91f3-a86554a6e761" containerName="istio-proxy"
Nov 01 06:17:44.387834 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.387783    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39efefc9-9afa-4664-91f3-a86554a6e761" containerName="istio-proxy"
Nov 01 06:17:44.387834 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.387820    5619 cpu_manager.go:414] "containerMap: removing container" podUID="bed3762c-2388-4654-9555-6c4cc3b817e3" containerName="payload-pre-processing"
Nov 01 06:17:44.387834 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.387829    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed3762c-2388-4654-9555-6c4cc3b817e3" containerName="payload-pre-processing"
Nov 01 06:17:44.388084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.387843    5619 cpu_manager.go:414] "containerMap: removing container" podUID="3eb9ca80-f436-41b4-8354-ba7498f47580" containerName="discovery"
Nov 01 06:17:44.388084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.387851    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb9ca80-f436-41b4-8354-ba7498f47580" containerName="discovery"
Nov 01 06:17:44.388084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.387870    5619 cpu_manager.go:414] "containerMap: removing container" podUID="19041e40-ee3c-4e90-908e-e3072a30e985" containerName="maas-api"
Nov 01 06:17:44.388084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.387877    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19041e40-ee3c-4e90-908e-e3072a30e985" containerName="maas-api"
Nov 01 06:17:44.388084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.387899    5619 cpu_manager.go:414] "containerMap: removing container" podUID="e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" containerName="istio-proxy"
Nov 01 06:17:44.388084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.387905    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" containerName="istio-proxy"
Nov 01 06:17:44.388084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.387917    5619 cpu_manager.go:414] "containerMap: removing container" podUID="0e66b197-c227-414d-8fa8-8f39a228fabd" containerName="payload-processing"
Nov 01 06:17:44.388084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.387924    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e66b197-c227-414d-8fa8-8f39a228fabd" containerName="payload-processing"
Nov 01 06:17:44.388084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.388021    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="39efefc9-9afa-4664-91f3-a86554a6e761" containerName="istio-proxy"
Nov 01 06:17:44.388084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.388032    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" containerName="istio-proxy"
Nov 01 06:17:44.388084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.388040    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e66b197-c227-414d-8fa8-8f39a228fabd" containerName="payload-processing"
Nov 01 06:17:44.388084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.388052    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed3762c-2388-4654-9555-6c4cc3b817e3" containerName="payload-pre-processing"
Nov 01 06:17:44.388084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.388062    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" containerName="istio-proxy"
Nov 01 06:17:44.388084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.388073    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="231e12b3-f2c4-4633-8a9b-4dcfc8c34206" containerName="istio-proxy"
Nov 01 06:17:44.388084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.388082    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb9ca80-f436-41b4-8354-ba7498f47580" containerName="discovery"
Nov 01 06:17:44.388519 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.388092    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="074ba060-2a84-479d-9d8b-92987fb010e9" containerName="istio-proxy"
Nov 01 06:17:44.388519 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.388101    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="d21d21dc-79f2-40c1-b701-e75e87bd144c" containerName="maas-api"
Nov 01 06:17:44.388519 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.388108    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="19041e40-ee3c-4e90-908e-e3072a30e985" containerName="maas-api"
Nov 01 06:17:44.388519 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.388119    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="23561314-57a9-4def-a052-32c23ccdd1f0" containerName="istio-proxy"
Nov 01 06:17:44.388519 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.388128    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="e17e1eaf-fae0-49af-bfe9-8fe6b04bc163" containerName="istio-proxy"
Nov 01 06:17:44.392984 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.392964    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.398705 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.398679    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:44.448227 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.448191    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-99e79c-openshift-default-dockercfg-bwthx\""
Nov 01 06:17:44.475191 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.475148    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182e059b-08d3-4ea4-9a37-3bdc44581625-istiod-ca-cert\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.475365 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.475278    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-credential-socket\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.475365 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.475343    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-podinfo\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.475501 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.475423    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-token\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.475557 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.475530    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-workload-certs\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.475610 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.475583    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2hvt\" (UniqueName: \"kubernetes.io/projected/182e059b-08d3-4ea4-9a37-3bdc44581625-kube-api-access-z2hvt\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.475710 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.475691    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-workload-socket\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.475784 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.475749    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-envoy\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.475838 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.475781    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-data\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.501414 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.501382    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"]
Nov 01 06:17:44.582345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.578997    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2hvt\" (UniqueName: \"kubernetes.io/projected/182e059b-08d3-4ea4-9a37-3bdc44581625-kube-api-access-z2hvt\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.582345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.579062    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-workload-socket\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.582345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.579117    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-envoy\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.582345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.579148    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-data\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.582345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.579210    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182e059b-08d3-4ea4-9a37-3bdc44581625-istiod-ca-cert\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.582345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.579285    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-credential-socket\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.582345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.579315    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-podinfo\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.582345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.579390    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-token\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.582345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.579481    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-workload-certs\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.582345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.579930    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-workload-certs\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.582345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.580599    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-workload-socket\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.594351 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.587371    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-credential-socket\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.594351 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.587641    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-data\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.594351 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.588254    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182e059b-08d3-4ea4-9a37-3bdc44581625-istiod-ca-cert\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.594351 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.591842    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:44.604367 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.603516    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-envoy\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.626440 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.626406    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-podinfo\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.641982 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.641919    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2hvt\" (UniqueName: \"kubernetes.io/projected/182e059b-08d3-4ea4-9a37-3bdc44581625-kube-api-access-z2hvt\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.653246 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.653220    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-token\") pod \"e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") " pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.706101 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.706066    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:44.742421 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.742375    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:44.831314 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.831276    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"]
Nov 01 06:17:44.831514 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.831473    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv" podUID="39ae3e00-d79f-473e-ae51-c5b5029f5353" containerName="istio-proxy" containerID="cri-o://5941754270fab4969e3f000da9d354c983bf67936511f09cc2920a305361fe12" gracePeriod=30
Nov 01 06:17:44.896899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:44.896818    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:45.159574 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:45.159548    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"]
Nov 01 06:17:45.161083 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:17:45.161049    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182e059b_08d3_4ea4_9a37_3bdc44581625.slice/crio-00cb8d76224c4ab5234db223478196294a3acd500eb806a69b6d0f5b44cdc6f7 WatchSource:0}: Error finding container 00cb8d76224c4ab5234db223478196294a3acd500eb806a69b6d0f5b44cdc6f7: Status 404 returned error can't find the container with id 00cb8d76224c4ab5234db223478196294a3acd500eb806a69b6d0f5b44cdc6f7
Nov 01 06:17:45.163549 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:45.163516    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:17:45.163648 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:45.163583    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:17:45.163648 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:45.163613    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:17:45.440237 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:45.440131    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:45.549774 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:45.549740    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb" event={"ID":"182e059b-08d3-4ea4-9a37-3bdc44581625","Type":"ContainerStarted","Data":"60da1d2de41805be3601e18106624ba7eea709026d5469985898569c8091de49"}
Nov 01 06:17:45.549945 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:45.549791    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb" event={"ID":"182e059b-08d3-4ea4-9a37-3bdc44581625","Type":"ContainerStarted","Data":"00cb8d76224c4ab5234db223478196294a3acd500eb806a69b6d0f5b44cdc6f7"}
Nov 01 06:17:45.672473 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:45.672421    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb" podStartSLOduration=1.672404242 podStartE2EDuration="1.672404242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:17:45.671781333 +0000 UTC m=+1902.882604860" watchObservedRunningTime="2026-11-01 06:17:45.672404242 +0000 UTC m=+1902.883227814"
Nov 01 06:17:45.706660 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:45.706594    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:46.640515 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:46.640478    5619 patch_prober.go:29] interesting pod/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:17:46.640867 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:46.640531    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv" podUID="39ae3e00-d79f-473e-ae51-c5b5029f5353" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:17:46.712154 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:46.712125    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:47.490998 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:47.490960    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:47.560144 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:47.560105    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:47.561240 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:47.561220    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:17:47.687129 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:47.687094    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:48.410915 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:48.410876    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:48.641161 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:48.641128    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:48.794909 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:48.794816    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:48.981060 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:48.981027    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:50.097718 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.097661    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:17:50.259065 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.259031    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-workload-socket\") pod \"39ae3e00-d79f-473e-ae51-c5b5029f5353\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") "
Nov 01 06:17:50.259065 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.259070    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39ae3e00-d79f-473e-ae51-c5b5029f5353-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ae3e00-d79f-473e-ae51-c5b5029f5353-istiod-ca-cert\") pod \"39ae3e00-d79f-473e-ae51-c5b5029f5353\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") "
Nov 01 06:17:50.259292 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.259096    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-token\" (UniqueName: \"kubernetes.io/projected/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-token\") pod \"39ae3e00-d79f-473e-ae51-c5b5029f5353\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") "
Nov 01 06:17:50.259292 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.259128    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-workload-certs\") pod \"39ae3e00-d79f-473e-ae51-c5b5029f5353\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") "
Nov 01 06:17:50.259292 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.259161    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ae3e00-d79f-473e-ae51-c5b5029f5353-kube-api-access-qtmqw\" (UniqueName: \"kubernetes.io/projected/39ae3e00-d79f-473e-ae51-c5b5029f5353-kube-api-access-qtmqw\") pod \"39ae3e00-d79f-473e-ae51-c5b5029f5353\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") "
Nov 01 06:17:50.259540 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.259365    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-data\") pod \"39ae3e00-d79f-473e-ae51-c5b5029f5353\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") "
Nov 01 06:17:50.259540 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.259385    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-workload-socket" pod "39ae3e00-d79f-473e-ae51-c5b5029f5353" (UID: "39ae3e00-d79f-473e-ae51-c5b5029f5353"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:17:50.259540 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.259427    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-podinfo\") pod \"39ae3e00-d79f-473e-ae51-c5b5029f5353\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") "
Nov 01 06:17:50.259540 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.259507    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-workload-certs" pod "39ae3e00-d79f-473e-ae51-c5b5029f5353" (UID: "39ae3e00-d79f-473e-ae51-c5b5029f5353"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:17:50.259540 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.259518    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-envoy\") pod \"39ae3e00-d79f-473e-ae51-c5b5029f5353\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") "
Nov 01 06:17:50.259540 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.259525    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ae3e00-d79f-473e-ae51-c5b5029f5353-istiod-ca-cert" pod "39ae3e00-d79f-473e-ae51-c5b5029f5353" (UID: "39ae3e00-d79f-473e-ae51-c5b5029f5353"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:17:50.259847 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.259571    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-credential-socket\") pod \"39ae3e00-d79f-473e-ae51-c5b5029f5353\" (UID: \"39ae3e00-d79f-473e-ae51-c5b5029f5353\") "
Nov 01 06:17:50.259847 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.259726    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-data" pod "39ae3e00-d79f-473e-ae51-c5b5029f5353" (UID: "39ae3e00-d79f-473e-ae51-c5b5029f5353"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:17:50.259935 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.259905    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-credential-socket" pod "39ae3e00-d79f-473e-ae51-c5b5029f5353" (UID: "39ae3e00-d79f-473e-ae51-c5b5029f5353"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:17:50.260102 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.260083    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ae3e00-d79f-473e-ae51-c5b5029f5353-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:50.260163 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.260110    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:50.260163 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.260125    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:50.260163 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.260142    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:50.260163 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.260156    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:50.261664 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.261636    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ae3e00-d79f-473e-ae51-c5b5029f5353-kube-api-access-qtmqw" pod "39ae3e00-d79f-473e-ae51-c5b5029f5353" (UID: "39ae3e00-d79f-473e-ae51-c5b5029f5353"). InnerVolumeSpecName "kube-api-access-qtmqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:17:50.261664 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.261654    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-token" pod "39ae3e00-d79f-473e-ae51-c5b5029f5353" (UID: "39ae3e00-d79f-473e-ae51-c5b5029f5353"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:17:50.261907 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.261883    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-envoy" pod "39ae3e00-d79f-473e-ae51-c5b5029f5353" (UID: "39ae3e00-d79f-473e-ae51-c5b5029f5353"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:17:50.261979 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.261947    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-podinfo" pod "39ae3e00-d79f-473e-ae51-c5b5029f5353" (UID: "39ae3e00-d79f-473e-ae51-c5b5029f5353"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:17:50.361567 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.361484    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:50.361567 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.361523    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:50.361567 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.361538    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ae3e00-d79f-473e-ae51-c5b5029f5353-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:50.361567 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.361552    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtmqw\" (UniqueName: \"kubernetes.io/projected/39ae3e00-d79f-473e-ae51-c5b5029f5353-kube-api-access-qtmqw\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:17:50.578209 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.578168    5619 generic.go:356] "Generic (PLEG): container finished" podID="39ae3e00-d79f-473e-ae51-c5b5029f5353" containerID="5941754270fab4969e3f000da9d354c983bf67936511f09cc2920a305361fe12" exitCode=0
Nov 01 06:17:50.578389 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.578230    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"
Nov 01 06:17:50.578389 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.578233    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv" event={"ID":"39ae3e00-d79f-473e-ae51-c5b5029f5353","Type":"ContainerDied","Data":"5941754270fab4969e3f000da9d354c983bf67936511f09cc2920a305361fe12"}
Nov 01 06:17:50.578389 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.578295    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv" event={"ID":"39ae3e00-d79f-473e-ae51-c5b5029f5353","Type":"ContainerDied","Data":"8e3e4de014edfe309bf5478d0247908bc759f482e6cd78dcb1b2385f10109e64"}
Nov 01 06:17:50.578389 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.578342    5619 scope.go:122] "RemoveContainer" containerID="5941754270fab4969e3f000da9d354c983bf67936511f09cc2920a305361fe12"
Nov 01 06:17:50.589590 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.589573    5619 scope.go:122] "RemoveContainer" containerID="5e37f4871a4556972d0d93b79f58144305d7bbe0a03343b70e755935605b7b34"
Nov 01 06:17:50.600028 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.600011    5619 scope.go:122] "RemoveContainer" containerID="5941754270fab4969e3f000da9d354c983bf67936511f09cc2920a305361fe12"
Nov 01 06:17:50.600351 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:17:50.600306    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5941754270fab4969e3f000da9d354c983bf67936511f09cc2920a305361fe12\": container with ID starting with 5941754270fab4969e3f000da9d354c983bf67936511f09cc2920a305361fe12 not found: ID does not exist" containerID="5941754270fab4969e3f000da9d354c983bf67936511f09cc2920a305361fe12"
Nov 01 06:17:50.600447 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.600360    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5941754270fab4969e3f000da9d354c983bf67936511f09cc2920a305361fe12"} err="failed to get container status \"5941754270fab4969e3f000da9d354c983bf67936511f09cc2920a305361fe12\": rpc error: code = NotFound desc = could not find container \"5941754270fab4969e3f000da9d354c983bf67936511f09cc2920a305361fe12\": container with ID starting with 5941754270fab4969e3f000da9d354c983bf67936511f09cc2920a305361fe12 not found: ID does not exist"
Nov 01 06:17:50.600447 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.600378    5619 scope.go:122] "RemoveContainer" containerID="5e37f4871a4556972d0d93b79f58144305d7bbe0a03343b70e755935605b7b34"
Nov 01 06:17:50.600680 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:17:50.600645    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e37f4871a4556972d0d93b79f58144305d7bbe0a03343b70e755935605b7b34\": container with ID starting with 5e37f4871a4556972d0d93b79f58144305d7bbe0a03343b70e755935605b7b34 not found: ID does not exist" containerID="5e37f4871a4556972d0d93b79f58144305d7bbe0a03343b70e755935605b7b34"
Nov 01 06:17:50.600806 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.600685    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e37f4871a4556972d0d93b79f58144305d7bbe0a03343b70e755935605b7b34"} err="failed to get container status \"5e37f4871a4556972d0d93b79f58144305d7bbe0a03343b70e755935605b7b34\": rpc error: code = NotFound desc = could not find container \"5e37f4871a4556972d0d93b79f58144305d7bbe0a03343b70e755935605b7b34\": container with ID starting with 5e37f4871a4556972d0d93b79f58144305d7bbe0a03343b70e755935605b7b34 not found: ID does not exist"
Nov 01 06:17:50.656078 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.656003    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:50.746400 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.746361    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"]
Nov 01 06:17:50.797204 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.797165    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-ec5596-openshift-default-69df67b7df-276tv"]
Nov 01 06:17:50.962850 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:50.962760    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ae3e00-d79f-473e-ae51-c5b5029f5353" path="/var/lib/kubelet/pods/39ae3e00-d79f-473e-ae51-c5b5029f5353/volumes"
Nov 01 06:17:51.769127 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:51.769078    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:51.960987 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:51.960951    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:52.184418 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.184384    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr"]
Nov 01 06:17:52.184879 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.184866    5619 cpu_manager.go:414] "containerMap: removing container" podUID="39ae3e00-d79f-473e-ae51-c5b5029f5353" containerName="istio-proxy"
Nov 01 06:17:52.184925 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.184880    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ae3e00-d79f-473e-ae51-c5b5029f5353" containerName="istio-proxy"
Nov 01 06:17:52.184925 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.184915    5619 cpu_manager.go:414] "containerMap: removing container" podUID="7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" containerName="istio-proxy"
Nov 01 06:17:52.184925 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.184920    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b3e9603-cafa-487a-9b7b-9b6b5ea7ebea" containerName="istio-proxy"
Nov 01 06:17:52.185015 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.184995    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ae3e00-d79f-473e-ae51-c5b5029f5353" containerName="istio-proxy"
Nov 01 06:17:52.185015 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.185005    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ae3e00-d79f-473e-ae51-c5b5029f5353" containerName="istio-proxy"
Nov 01 06:17:52.189722 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.189702    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr"
Nov 01 06:17:52.244157 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.243802    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-99e79c\""
Nov 01 06:17:52.244157 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.243805    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-99e79c-dockercfg-ftvl6\""
Nov 01 06:17:52.327390 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.300216    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr"]
Nov 01 06:17:52.327390 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.324416    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drlsh\" (UniqueName: \"kubernetes.io/projected/df8c3448-d616-414e-8e09-d39e82928731-kube-api-access-drlsh\") pod \"payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr\" (UID: \"df8c3448-d616-414e-8e09-d39e82928731\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr"
Nov 01 06:17:52.327390 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.326785    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df8c3448-d616-414e-8e09-d39e82928731-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr\" (UID: \"df8c3448-d616-414e-8e09-d39e82928731\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr"
Nov 01 06:17:52.339705 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.339646    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6"]
Nov 01 06:17:52.340508 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.340462    5619 cpu_manager.go:414] "containerMap: removing container" podUID="39ae3e00-d79f-473e-ae51-c5b5029f5353" containerName="istio-proxy"
Nov 01 06:17:52.340718 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.340692    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ae3e00-d79f-473e-ae51-c5b5029f5353" containerName="istio-proxy"
Nov 01 06:17:52.360951 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.360916    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6"
Nov 01 06:17:52.395307 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.395274    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6"]
Nov 01 06:17:52.428361 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.428311    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drlsh\" (UniqueName: \"kubernetes.io/projected/df8c3448-d616-414e-8e09-d39e82928731-kube-api-access-drlsh\") pod \"payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr\" (UID: \"df8c3448-d616-414e-8e09-d39e82928731\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr"
Nov 01 06:17:52.428532 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.428380    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ec112e9-286e-42f5-be40-f6954033585c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6\" (UID: \"7ec112e9-286e-42f5-be40-f6954033585c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6"
Nov 01 06:17:52.428532 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.428423    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt47x\" (UniqueName: \"kubernetes.io/projected/7ec112e9-286e-42f5-be40-f6954033585c-kube-api-access-rt47x\") pod \"payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6\" (UID: \"7ec112e9-286e-42f5-be40-f6954033585c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6"
Nov 01 06:17:52.428532 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.428460    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df8c3448-d616-414e-8e09-d39e82928731-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr\" (UID: \"df8c3448-d616-414e-8e09-d39e82928731\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr"
Nov 01 06:17:52.430102 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.430063    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df8c3448-d616-414e-8e09-d39e82928731-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr\" (UID: \"df8c3448-d616-414e-8e09-d39e82928731\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr"
Nov 01 06:17:52.490516 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.490442    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drlsh\" (UniqueName: \"kubernetes.io/projected/df8c3448-d616-414e-8e09-d39e82928731-kube-api-access-drlsh\") pod \"payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr\" (UID: \"df8c3448-d616-414e-8e09-d39e82928731\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr"
Nov 01 06:17:52.504132 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.504107    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr"
Nov 01 06:17:52.530219 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.530187    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ec112e9-286e-42f5-be40-f6954033585c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6\" (UID: \"7ec112e9-286e-42f5-be40-f6954033585c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6"
Nov 01 06:17:52.530360 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.530250    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt47x\" (UniqueName: \"kubernetes.io/projected/7ec112e9-286e-42f5-be40-f6954033585c-kube-api-access-rt47x\") pod \"payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6\" (UID: \"7ec112e9-286e-42f5-be40-f6954033585c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6"
Nov 01 06:17:52.531077 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.531051    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ec112e9-286e-42f5-be40-f6954033585c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6\" (UID: \"7ec112e9-286e-42f5-be40-f6954033585c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6"
Nov 01 06:17:52.596889 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.596827    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt47x\" (UniqueName: \"kubernetes.io/projected/7ec112e9-286e-42f5-be40-f6954033585c-kube-api-access-rt47x\") pod \"payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6\" (UID: \"7ec112e9-286e-42f5-be40-f6954033585c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6"
Nov 01 06:17:52.677780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.677750    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6"
Nov 01 06:17:52.879978 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:52.879950    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr"]
Nov 01 06:17:52.882075 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:17:52.882047    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf8c3448_d616_414e_8e09_d39e82928731.slice/crio-9ed05961134c6306cddf49213930b26006a6f3c2a30c1a581570b9af174aeef1 WatchSource:0}: Error finding container 9ed05961134c6306cddf49213930b26006a6f3c2a30c1a581570b9af174aeef1: Status 404 returned error can't find the container with id 9ed05961134c6306cddf49213930b26006a6f3c2a30c1a581570b9af174aeef1
Nov 01 06:17:53.054940 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:53.054903    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6"]
Nov 01 06:17:53.056630 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:17:53.056599    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec112e9_286e_42f5_be40_f6954033585c.slice/crio-85634efbc0c861df86a58021a34ffbc5a728c768dbc685d334f646f81869cb9e WatchSource:0}: Error finding container 85634efbc0c861df86a58021a34ffbc5a728c768dbc685d334f646f81869cb9e: Status 404 returned error can't find the container with id 85634efbc0c861df86a58021a34ffbc5a728c768dbc685d334f646f81869cb9e
Nov 01 06:17:53.602779 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:53.602745    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6" event={"ID":"7ec112e9-286e-42f5-be40-f6954033585c","Type":"ContainerStarted","Data":"8528ac496f37e78c700a795f4c2e4b373e2fc0963ebaa2fb17f79307cd4af4c8"}
Nov 01 06:17:53.602985 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:53.602798    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6" event={"ID":"7ec112e9-286e-42f5-be40-f6954033585c","Type":"ContainerStarted","Data":"85634efbc0c861df86a58021a34ffbc5a728c768dbc685d334f646f81869cb9e"}
Nov 01 06:17:53.602985 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:53.602846    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6"
Nov 01 06:17:53.604634 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:53.604608    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr" event={"ID":"df8c3448-d616-414e-8e09-d39e82928731","Type":"ContainerStarted","Data":"ae6f3be15d6d0078921b00421a388ba4b2b28a394a747f4c9ddc07bd21e7a5db"}
Nov 01 06:17:53.604754 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:53.604648    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr" event={"ID":"df8c3448-d616-414e-8e09-d39e82928731","Type":"ContainerStarted","Data":"9ed05961134c6306cddf49213930b26006a6f3c2a30c1a581570b9af174aeef1"}
Nov 01 06:17:53.604754 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:53.604689    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr"
Nov 01 06:17:53.722043 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:53.721994    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6" podStartSLOduration=1.721980474 podStartE2EDuration="1.721980474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:17:53.719901549 +0000 UTC m=+1910.930725057" watchObservedRunningTime="2026-11-01 06:17:53.721980474 +0000 UTC m=+1910.932803983"
Nov 01 06:17:53.842592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:53.842525    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr" podStartSLOduration=1.842505958 podStartE2EDuration="1.842505958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:17:53.838632034 +0000 UTC m=+1911.049455544" watchObservedRunningTime="2026-11-01 06:17:53.842505958 +0000 UTC m=+1911.053329468"
Nov 01 06:17:54.857378 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:54.857339    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:55.654258 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:55.654174    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:55.879390 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:17:55.879353    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:17:55.932010 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:55.931921    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:17:58.223064 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:17:58.223022    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:00.377582 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:00.377547    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:04.127861 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:04.127818    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:04.285597 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:04.285566    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:04.454938 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:04.454852    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:04.613745 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:04.613717    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr"
Nov 01 06:18:04.613915 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:04.613780    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6"
Nov 01 06:18:04.979004 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:04.978972    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"]
Nov 01 06:18:04.984216 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:04.984196    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.063016 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.062974    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-envoy\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.066445 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.066408    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93542be2-aa8a-4d1f-ade6-33dd834c5758-istiod-ca-cert\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.066695 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.066678    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-credential-socket\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.066840 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.066815    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-token\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.066988 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.066891    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt4hr\" (UniqueName: \"kubernetes.io/projected/93542be2-aa8a-4d1f-ade6-33dd834c5758-kube-api-access-vt4hr\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.067055 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.067031    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-podinfo\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.067121 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.067076    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-workload-certs\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.067295 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.067276    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-workload-socket\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.067560 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.067317    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-data\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.082678 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.082650    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"]
Nov 01 06:18:05.110348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.107491    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2a4cda2f-openshift-default-dockercfg-g6plg\""
Nov 01 06:18:05.128416 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.128379    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:05.168896 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.168866    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-workload-socket\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.169081 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.168903    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-data\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.169081 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.168925    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-envoy\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.169081 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.169056    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93542be2-aa8a-4d1f-ade6-33dd834c5758-istiod-ca-cert\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.169252 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.169111    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-credential-socket\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.169252 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.169139    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-token\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.169252 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.169206    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt4hr\" (UniqueName: \"kubernetes.io/projected/93542be2-aa8a-4d1f-ade6-33dd834c5758-kube-api-access-vt4hr\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.169443 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.169358    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-podinfo\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.169443 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.169366    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-data\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.169443 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.169411    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-workload-certs\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.169443 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.169421    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-credential-socket\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.169655 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.169530    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-workload-socket\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.169802 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.169772    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-workload-certs\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.170239 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.170217    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93542be2-aa8a-4d1f-ade6-33dd834c5758-istiod-ca-cert\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.171528 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.171503    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-envoy\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.171739 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.171717    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-podinfo\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.232061 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.231987    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-token\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.232908 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.232881    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt4hr\" (UniqueName: \"kubernetes.io/projected/93542be2-aa8a-4d1f-ade6-33dd834c5758-kube-api-access-vt4hr\") pod \"e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") " pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.298432 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.298398    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:05.322696 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.322660    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:05.473142 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.473105    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:05.738701 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.738672    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"]
Nov 01 06:18:05.741798 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:18:05.741768    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93542be2_aa8a_4d1f_ade6_33dd834c5758.slice/crio-f94d5402fcd8b2a1daa107890c052bd0628afea8446617225dc7763fb22faca1 WatchSource:0}: Error finding container f94d5402fcd8b2a1daa107890c052bd0628afea8446617225dc7763fb22faca1: Status 404 returned error can't find the container with id f94d5402fcd8b2a1daa107890c052bd0628afea8446617225dc7763fb22faca1
Nov 01 06:18:05.744557 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.744515    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:18:05.744673 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.744623    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:18:05.744733 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:05.744668    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:18:06.682393 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:06.682359    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt" event={"ID":"93542be2-aa8a-4d1f-ade6-33dd834c5758","Type":"ContainerStarted","Data":"66bc50bf8e637149ede5b711b34b7c833d6e4393672d06bcd7c1b07c37406877"}
Nov 01 06:18:06.682858 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:06.682409    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt" event={"ID":"93542be2-aa8a-4d1f-ade6-33dd834c5758","Type":"ContainerStarted","Data":"f94d5402fcd8b2a1daa107890c052bd0628afea8446617225dc7763fb22faca1"}
Nov 01 06:18:06.856190 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:06.856129    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt" podStartSLOduration=2.856113308 podStartE2EDuration="2.856113308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:18:06.853903407 +0000 UTC m=+1924.064726916" watchObservedRunningTime="2026-11-01 06:18:06.856113308 +0000 UTC m=+1924.066936816"
Nov 01 06:18:07.299053 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:07.299021    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:07.304374 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:07.304347    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:07.688383 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:07.688349    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:07.689382 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:07.689358    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:18:08.973426 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:08.973393    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:09.135865 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:09.135834    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:09.287677 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:09.287597    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:10.103242 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:10.103205    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:12.181296 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:12.181237    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:12.352630 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:12.352597    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:12.590428 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:12.590403    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2"]
Nov 01 06:18:12.594440 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:12.594423    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2"
Nov 01 06:18:12.656995 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:12.656961    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe3528c-3bc2-447c-93e5-ce82909e5fb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2\" (UID: \"ffe3528c-3bc2-447c-93e5-ce82909e5fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2"
Nov 01 06:18:12.657155 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:12.657031    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xlls\" (UniqueName: \"kubernetes.io/projected/ffe3528c-3bc2-447c-93e5-ce82909e5fb1-kube-api-access-9xlls\") pod \"payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2\" (UID: \"ffe3528c-3bc2-447c-93e5-ce82909e5fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2"
Nov 01 06:18:12.702666 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:12.702632    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2a4cda2f-dockercfg-xnb44\""
Nov 01 06:18:12.703571 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:12.703543    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2a4cda2f\""
Nov 01 06:18:12.766107 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:12.763198    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe3528c-3bc2-447c-93e5-ce82909e5fb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2\" (UID: \"ffe3528c-3bc2-447c-93e5-ce82909e5fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2"
Nov 01 06:18:12.766107 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:12.763276    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xlls\" (UniqueName: \"kubernetes.io/projected/ffe3528c-3bc2-447c-93e5-ce82909e5fb1-kube-api-access-9xlls\") pod \"payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2\" (UID: \"ffe3528c-3bc2-447c-93e5-ce82909e5fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2"
Nov 01 06:18:12.766107 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:12.764883    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe3528c-3bc2-447c-93e5-ce82909e5fb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2\" (UID: \"ffe3528c-3bc2-447c-93e5-ce82909e5fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2"
Nov 01 06:18:12.783038 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:12.783004    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2"]
Nov 01 06:18:12.788349 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:12.788300    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v"]
Nov 01 06:18:12.792910 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:12.792858    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v"
Nov 01 06:18:12.806844 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:12.806737    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v"]
Nov 01 06:18:12.825758 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:12.825732    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xlls\" (UniqueName: \"kubernetes.io/projected/ffe3528c-3bc2-447c-93e5-ce82909e5fb1-kube-api-access-9xlls\") pod \"payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2\" (UID: \"ffe3528c-3bc2-447c-93e5-ce82909e5fb1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2"
Nov 01 06:18:12.864861 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:12.864780    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whbqd\" (UniqueName: \"kubernetes.io/projected/4eba5cde-05e2-4926-89c6-11231b9060ea-kube-api-access-whbqd\") pod \"payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v\" (UID: \"4eba5cde-05e2-4926-89c6-11231b9060ea\") " pod="openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v"
Nov 01 06:18:12.864861 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:12.864836    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eba5cde-05e2-4926-89c6-11231b9060ea-plugins-config-volume\") pod \"payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v\" (UID: \"4eba5cde-05e2-4926-89c6-11231b9060ea\") " pod="openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v"
Nov 01 06:18:12.909937 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:12.909910    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2"
Nov 01 06:18:12.971756 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:12.971626    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whbqd\" (UniqueName: \"kubernetes.io/projected/4eba5cde-05e2-4926-89c6-11231b9060ea-kube-api-access-whbqd\") pod \"payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v\" (UID: \"4eba5cde-05e2-4926-89c6-11231b9060ea\") " pod="openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v"
Nov 01 06:18:12.971756 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:12.971713    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eba5cde-05e2-4926-89c6-11231b9060ea-plugins-config-volume\") pod \"payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v\" (UID: \"4eba5cde-05e2-4926-89c6-11231b9060ea\") " pod="openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v"
Nov 01 06:18:12.974227 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:12.974191    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eba5cde-05e2-4926-89c6-11231b9060ea-plugins-config-volume\") pod \"payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v\" (UID: \"4eba5cde-05e2-4926-89c6-11231b9060ea\") " pod="openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v"
Nov 01 06:18:13.034805 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:13.034765    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whbqd\" (UniqueName: \"kubernetes.io/projected/4eba5cde-05e2-4926-89c6-11231b9060ea-kube-api-access-whbqd\") pod \"payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v\" (UID: \"4eba5cde-05e2-4926-89c6-11231b9060ea\") " pod="openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v"
Nov 01 06:18:13.107458 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:13.107429    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v"
Nov 01 06:18:13.300507 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:13.300475    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2"]
Nov 01 06:18:13.302416 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:18:13.302388    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe3528c_3bc2_447c_93e5_ce82909e5fb1.slice/crio-3033647ceb5fe6fd432d9449033481d59947446308d6f2a876bd0cebc59e2b93 WatchSource:0}: Error finding container 3033647ceb5fe6fd432d9449033481d59947446308d6f2a876bd0cebc59e2b93: Status 404 returned error can't find the container with id 3033647ceb5fe6fd432d9449033481d59947446308d6f2a876bd0cebc59e2b93
Nov 01 06:18:13.485458 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:13.485427    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v"]
Nov 01 06:18:13.487026 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:18:13.486990    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eba5cde_05e2_4926_89c6_11231b9060ea.slice/crio-48a307d52e5be665bb62346378c12cb6b58a5078d7607da8d3e6a3d0c6abfba3 WatchSource:0}: Error finding container 48a307d52e5be665bb62346378c12cb6b58a5078d7607da8d3e6a3d0c6abfba3: Status 404 returned error can't find the container with id 48a307d52e5be665bb62346378c12cb6b58a5078d7607da8d3e6a3d0c6abfba3
Nov 01 06:18:13.788453 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:13.788352    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v" event={"ID":"4eba5cde-05e2-4926-89c6-11231b9060ea","Type":"ContainerStarted","Data":"ac85663bcc83222502a52312e9f8ed92e0a328271da8c9af77cc283e54eaa119"}
Nov 01 06:18:13.788453 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:13.788414    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v" event={"ID":"4eba5cde-05e2-4926-89c6-11231b9060ea","Type":"ContainerStarted","Data":"48a307d52e5be665bb62346378c12cb6b58a5078d7607da8d3e6a3d0c6abfba3"}
Nov 01 06:18:13.788675 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:13.788534    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v"
Nov 01 06:18:13.790046 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:13.790014    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2" event={"ID":"ffe3528c-3bc2-447c-93e5-ce82909e5fb1","Type":"ContainerStarted","Data":"5ba127f81be4b1c0fdd0af7dbcce149f7909504fb921be11b40e4a11c93838c4"}
Nov 01 06:18:13.790180 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:13.790070    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2" event={"ID":"ffe3528c-3bc2-447c-93e5-ce82909e5fb1","Type":"ContainerStarted","Data":"3033647ceb5fe6fd432d9449033481d59947446308d6f2a876bd0cebc59e2b93"}
Nov 01 06:18:13.790180 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:13.790107    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2"
Nov 01 06:18:13.907233 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:13.907168    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v" podStartSLOduration=1.9071487889999998 podStartE2EDuration="1.907148789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:18:13.905483912 +0000 UTC m=+1931.116307421" watchObservedRunningTime="2026-11-01 06:18:13.907148789 +0000 UTC m=+1931.117972300"
Nov 01 06:18:14.025589 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:14.025525    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2" podStartSLOduration=2.025505684 podStartE2EDuration="2.025505684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:18:14.023179272 +0000 UTC m=+1931.234002819" watchObservedRunningTime="2026-11-01 06:18:14.025505684 +0000 UTC m=+1931.236329194"
Nov 01 06:18:15.259740 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:15.259681    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:19.978917 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:19.978881    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:22.421767 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:22.421726    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:24.704479 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:24.704444    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:24.798368 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:24.798336    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2"
Nov 01 06:18:24.798539 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:24.798410    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v"
Nov 01 06:18:25.215059 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:25.215024    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"]
Nov 01 06:18:25.215373 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:25.215305    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl" podUID="6c53fe0e-52d4-48ca-b508-96aa30adea55" containerName="istio-proxy" containerID="cri-o://95eaa0df8c1e7738e25338365ebd98c8a2e1a4a680bf1dd564add80e4d53fc21" gracePeriod=30
Nov 01 06:18:25.880041 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:18:25.879997    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:18:26.173035 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:26.172954    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:18:26.173035 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:26.173019    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl" podUID="6c53fe0e-52d4-48ca-b508-96aa30adea55" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:18:27.228153 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:27.228120    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:29.404235 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:29.404194    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:30.461752 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.461726    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:18:30.470140 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.470114    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-envoy\") pod \"6c53fe0e-52d4-48ca-b508-96aa30adea55\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") "
Nov 01 06:18:30.470280 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.470195    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-workload-certs\") pod \"6c53fe0e-52d4-48ca-b508-96aa30adea55\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") "
Nov 01 06:18:30.470280 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.470217    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-workload-socket\") pod \"6c53fe0e-52d4-48ca-b508-96aa30adea55\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") "
Nov 01 06:18:30.470280 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.470237    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c53fe0e-52d4-48ca-b508-96aa30adea55-kube-api-access-hbhdr\" (UniqueName: \"kubernetes.io/projected/6c53fe0e-52d4-48ca-b508-96aa30adea55-kube-api-access-hbhdr\") pod \"6c53fe0e-52d4-48ca-b508-96aa30adea55\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") "
Nov 01 06:18:30.470280 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.470266    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-data\") pod \"6c53fe0e-52d4-48ca-b508-96aa30adea55\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") "
Nov 01 06:18:30.470531 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.470401    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-token\" (UniqueName: \"kubernetes.io/projected/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-token\") pod \"6c53fe0e-52d4-48ca-b508-96aa30adea55\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") "
Nov 01 06:18:30.470531 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.470431    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-credential-socket\") pod \"6c53fe0e-52d4-48ca-b508-96aa30adea55\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") "
Nov 01 06:18:30.470531 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.470467    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c53fe0e-52d4-48ca-b508-96aa30adea55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c53fe0e-52d4-48ca-b508-96aa30adea55-istiod-ca-cert\") pod \"6c53fe0e-52d4-48ca-b508-96aa30adea55\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") "
Nov 01 06:18:30.470531 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.470513    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-podinfo\") pod \"6c53fe0e-52d4-48ca-b508-96aa30adea55\" (UID: \"6c53fe0e-52d4-48ca-b508-96aa30adea55\") "
Nov 01 06:18:30.470757 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.470559    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-workload-certs" pod "6c53fe0e-52d4-48ca-b508-96aa30adea55" (UID: "6c53fe0e-52d4-48ca-b508-96aa30adea55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:18:30.470757 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.470571    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-workload-socket" pod "6c53fe0e-52d4-48ca-b508-96aa30adea55" (UID: "6c53fe0e-52d4-48ca-b508-96aa30adea55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:18:30.470757 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.470697    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-credential-socket" pod "6c53fe0e-52d4-48ca-b508-96aa30adea55" (UID: "6c53fe0e-52d4-48ca-b508-96aa30adea55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:18:30.470918 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.470801    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-data" pod "6c53fe0e-52d4-48ca-b508-96aa30adea55" (UID: "6c53fe0e-52d4-48ca-b508-96aa30adea55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:18:30.470918 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.470886    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c53fe0e-52d4-48ca-b508-96aa30adea55-istiod-ca-cert" pod "6c53fe0e-52d4-48ca-b508-96aa30adea55" (UID: "6c53fe0e-52d4-48ca-b508-96aa30adea55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:18:30.471131 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.471111    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:30.471256 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.471134    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:30.471256 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.471149    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:30.471256 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.471162    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:30.471256 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.471173    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c53fe0e-52d4-48ca-b508-96aa30adea55-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:30.472463 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.472438    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-envoy" pod "6c53fe0e-52d4-48ca-b508-96aa30adea55" (UID: "6c53fe0e-52d4-48ca-b508-96aa30adea55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:18:30.472922 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.472902    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-token" pod "6c53fe0e-52d4-48ca-b508-96aa30adea55" (UID: "6c53fe0e-52d4-48ca-b508-96aa30adea55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:18:30.473002 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.472944    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c53fe0e-52d4-48ca-b508-96aa30adea55-kube-api-access-hbhdr" pod "6c53fe0e-52d4-48ca-b508-96aa30adea55" (UID: "6c53fe0e-52d4-48ca-b508-96aa30adea55"). InnerVolumeSpecName "kube-api-access-hbhdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:18:30.473079 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.473059    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-podinfo" pod "6c53fe0e-52d4-48ca-b508-96aa30adea55" (UID: "6c53fe0e-52d4-48ca-b508-96aa30adea55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:18:30.572670 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.572599    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:30.572670 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.572625    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:30.572670 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.572635    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c53fe0e-52d4-48ca-b508-96aa30adea55-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:30.572670 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.572646    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbhdr\" (UniqueName: \"kubernetes.io/projected/6c53fe0e-52d4-48ca-b508-96aa30adea55-kube-api-access-hbhdr\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:30.887920 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.887873    5619 generic.go:356] "Generic (PLEG): container finished" podID="6c53fe0e-52d4-48ca-b508-96aa30adea55" containerID="95eaa0df8c1e7738e25338365ebd98c8a2e1a4a680bf1dd564add80e4d53fc21" exitCode=0
Nov 01 06:18:30.888103 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.887936    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"
Nov 01 06:18:30.888103 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.887934    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl" event={"ID":"6c53fe0e-52d4-48ca-b508-96aa30adea55","Type":"ContainerDied","Data":"95eaa0df8c1e7738e25338365ebd98c8a2e1a4a680bf1dd564add80e4d53fc21"}
Nov 01 06:18:30.888103 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.888055    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl" event={"ID":"6c53fe0e-52d4-48ca-b508-96aa30adea55","Type":"ContainerDied","Data":"abccc7c22ee0f54a41f76ec204804ef68ec0498a13a53d73b6a56874bc350ac4"}
Nov 01 06:18:30.888103 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.888078    5619 scope.go:122] "RemoveContainer" containerID="95eaa0df8c1e7738e25338365ebd98c8a2e1a4a680bf1dd564add80e4d53fc21"
Nov 01 06:18:30.898394 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.898380    5619 scope.go:122] "RemoveContainer" containerID="95eaa0df8c1e7738e25338365ebd98c8a2e1a4a680bf1dd564add80e4d53fc21"
Nov 01 06:18:30.898685 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:18:30.898669    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95eaa0df8c1e7738e25338365ebd98c8a2e1a4a680bf1dd564add80e4d53fc21\": container with ID starting with 95eaa0df8c1e7738e25338365ebd98c8a2e1a4a680bf1dd564add80e4d53fc21 not found: ID does not exist" containerID="95eaa0df8c1e7738e25338365ebd98c8a2e1a4a680bf1dd564add80e4d53fc21"
Nov 01 06:18:30.898724 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:30.898693    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95eaa0df8c1e7738e25338365ebd98c8a2e1a4a680bf1dd564add80e4d53fc21"} err="failed to get container status \"95eaa0df8c1e7738e25338365ebd98c8a2e1a4a680bf1dd564add80e4d53fc21\": rpc error: code = NotFound desc = could not find container \"95eaa0df8c1e7738e25338365ebd98c8a2e1a4a680bf1dd564add80e4d53fc21\": container with ID starting with 95eaa0df8c1e7738e25338365ebd98c8a2e1a4a680bf1dd564add80e4d53fc21 not found: ID does not exist"
Nov 01 06:18:31.056148 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:31.056109    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"]
Nov 01 06:18:31.108335 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:31.108288    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-qsjdl"]
Nov 01 06:18:32.962391 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:32.962361    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c53fe0e-52d4-48ca-b508-96aa30adea55" path="/var/lib/kubelet/pods/6c53fe0e-52d4-48ca-b508-96aa30adea55/volumes"
Nov 01 06:18:33.911893 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:33.911858    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"]
Nov 01 06:18:33.912463 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:33.912447    5619 cpu_manager.go:414] "containerMap: removing container" podUID="6c53fe0e-52d4-48ca-b508-96aa30adea55" containerName="istio-proxy"
Nov 01 06:18:33.912516 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:33.912465    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c53fe0e-52d4-48ca-b508-96aa30adea55" containerName="istio-proxy"
Nov 01 06:18:33.912603 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:33.912593    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c53fe0e-52d4-48ca-b508-96aa30adea55" containerName="istio-proxy"
Nov 01 06:18:33.917781 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:33.917763    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"
Nov 01 06:18:33.972452 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:33.972427    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-99e79c-serving-cert\""
Nov 01 06:18:34.013144 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:34.013104    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59x6p\" (UniqueName: \"kubernetes.io/projected/1e3999df-3132-435d-93b4-0ed000180c4d-kube-api-access-59x6p\") pod \"maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj\" (UID: \"1e3999df-3132-435d-93b4-0ed000180c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"
Nov 01 06:18:34.013278 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:34.013169    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3999df-3132-435d-93b4-0ed000180c4d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj\" (UID: \"1e3999df-3132-435d-93b4-0ed000180c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"
Nov 01 06:18:34.013370 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:34.013346    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e3999df-3132-435d-93b4-0ed000180c4d-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj\" (UID: \"1e3999df-3132-435d-93b4-0ed000180c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"
Nov 01 06:18:34.013413 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:34.013388    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e3999df-3132-435d-93b4-0ed000180c4d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj\" (UID: \"1e3999df-3132-435d-93b4-0ed000180c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"
Nov 01 06:18:34.025836 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:34.025801    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"]
Nov 01 06:18:34.105646 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:34.105610    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:34.114921 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:34.114900    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e3999df-3132-435d-93b4-0ed000180c4d-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj\" (UID: \"1e3999df-3132-435d-93b4-0ed000180c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"
Nov 01 06:18:34.114996 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:34.114935    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e3999df-3132-435d-93b4-0ed000180c4d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj\" (UID: \"1e3999df-3132-435d-93b4-0ed000180c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"
Nov 01 06:18:34.115102 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:34.115083    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59x6p\" (UniqueName: \"kubernetes.io/projected/1e3999df-3132-435d-93b4-0ed000180c4d-kube-api-access-59x6p\") pod \"maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj\" (UID: \"1e3999df-3132-435d-93b4-0ed000180c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"
Nov 01 06:18:34.115145 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:18:34.115108    5619 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-serving-cert: secret "maas-api-e2e-shared-b-w0-99e79c-serving-cert" not found
Nov 01 06:18:34.115185 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:34.115138    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3999df-3132-435d-93b4-0ed000180c4d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj\" (UID: \"1e3999df-3132-435d-93b4-0ed000180c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"
Nov 01 06:18:34.115185 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:18:34.115177    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e3999df-3132-435d-93b4-0ed000180c4d-maas-api-tls podName:1e3999df-3132-435d-93b4-0ed000180c4d nodeName:}" failed. No retries permitted until 2026-11-01 06:18:34.615157358 +0000 UTC m=+1951.825980845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1e3999df-3132-435d-93b4-0ed000180c4d-maas-api-tls") pod "maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj" (UID: "1e3999df-3132-435d-93b4-0ed000180c4d") : secret "maas-api-e2e-shared-b-w0-99e79c-serving-cert" not found
Nov 01 06:18:34.115765 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:34.115732    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3999df-3132-435d-93b4-0ed000180c4d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj\" (UID: \"1e3999df-3132-435d-93b4-0ed000180c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"
Nov 01 06:18:34.117347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:34.117309    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e3999df-3132-435d-93b4-0ed000180c4d-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj\" (UID: \"1e3999df-3132-435d-93b4-0ed000180c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"
Nov 01 06:18:34.178487 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:34.178377    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59x6p\" (UniqueName: \"kubernetes.io/projected/1e3999df-3132-435d-93b4-0ed000180c4d-kube-api-access-59x6p\") pod \"maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj\" (UID: \"1e3999df-3132-435d-93b4-0ed000180c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"
Nov 01 06:18:34.270850 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:34.270815    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:34.417263 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:34.417231    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:34.621566 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:34.621537    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e3999df-3132-435d-93b4-0ed000180c4d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj\" (UID: \"1e3999df-3132-435d-93b4-0ed000180c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"
Nov 01 06:18:34.623950 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:34.623929    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e3999df-3132-435d-93b4-0ed000180c4d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj\" (UID: \"1e3999df-3132-435d-93b4-0ed000180c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"
Nov 01 06:18:34.831979 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:34.831948    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"
Nov 01 06:18:35.250927 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:35.250895    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"]
Nov 01 06:18:35.252302 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:18:35.252280    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3999df_3132_435d_93b4_0ed000180c4d.slice/crio-a1452b06cefd2bd8ff4078e8bd6cfa1cf7a5c644779f26e6ce593f35dc79927f WatchSource:0}: Error finding container a1452b06cefd2bd8ff4078e8bd6cfa1cf7a5c644779f26e6ce593f35dc79927f: Status 404 returned error can't find the container with id a1452b06cefd2bd8ff4078e8bd6cfa1cf7a5c644779f26e6ce593f35dc79927f
Nov 01 06:18:35.253817 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:35.253798    5619 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 06:18:35.921778 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:35.921746    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj" event={"ID":"1e3999df-3132-435d-93b4-0ed000180c4d","Type":"ContainerStarted","Data":"a1452b06cefd2bd8ff4078e8bd6cfa1cf7a5c644779f26e6ce593f35dc79927f"}
Nov 01 06:18:36.929742 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:36.929708    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj" event={"ID":"1e3999df-3132-435d-93b4-0ed000180c4d","Type":"ContainerStarted","Data":"00bc0c5b50df72d72aadaa78e1a251246d6a940fb3e54678207cbc0422614d01"}
Nov 01 06:18:36.930225 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:36.929807    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"
Nov 01 06:18:37.049989 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:37.049933    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj" podStartSLOduration=3.189189428 podStartE2EDuration="4.0499181s" totalImagesPullingTime="860.728672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:18:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:18:37.048380891 +0000 UTC m=+1954.259204399" watchObservedRunningTime="2026-11-01 06:18:37.0499181 +0000 UTC m=+1954.260741609"
Nov 01 06:18:39.164512 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:39.164477    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:40.311497 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:40.311463    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"]
Nov 01 06:18:40.311892 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:40.311653    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2" podUID="dfbc6deb-ab23-4706-b60e-2eb924fbb008" containerName="istio-proxy" containerID="cri-o://c7f052316e9f6ca44265b3eb5ff0153e88ac7f51b5264fea791c0c13c6835ad1" gracePeriod=30
Nov 01 06:18:42.938807 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:42.938765    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj" podUID="1e3999df-3132-435d-93b4-0ed000180c4d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.89:8443/health\": dial tcp 10.133.0.89:8443: connect: connection refused"
Nov 01 06:18:43.987500 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:43.987460    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:45.582252 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.582182    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:18:45.647391 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.647360    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-data\") pod \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") "
Nov 01 06:18:45.647507 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.647458    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-token\" (UniqueName: \"kubernetes.io/projected/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-token\") pod \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") "
Nov 01 06:18:45.647555 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.647528    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-podinfo\") pod \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") "
Nov 01 06:18:45.647590 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.647578    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-workload-certs\") pod \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") "
Nov 01 06:18:45.647647 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.647608    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfbc6deb-ab23-4706-b60e-2eb924fbb008-kube-api-access-bqbxp\" (UniqueName: \"kubernetes.io/projected/dfbc6deb-ab23-4706-b60e-2eb924fbb008-kube-api-access-bqbxp\") pod \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") "
Nov 01 06:18:45.647647 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.647639    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-data" pod "dfbc6deb-ab23-4706-b60e-2eb924fbb008" (UID: "dfbc6deb-ab23-4706-b60e-2eb924fbb008"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:18:45.647750 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.647711    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-envoy\") pod \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") "
Nov 01 06:18:45.647805 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.647754    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-credential-socket\") pod \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") "
Nov 01 06:18:45.647862 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.647833    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-workload-socket\") pod \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") "
Nov 01 06:18:45.647862 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.647831    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-workload-certs" pod "dfbc6deb-ab23-4706-b60e-2eb924fbb008" (UID: "dfbc6deb-ab23-4706-b60e-2eb924fbb008"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:18:45.647970 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.647862    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istiod-ca-cert\") pod \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\" (UID: \"dfbc6deb-ab23-4706-b60e-2eb924fbb008\") "
Nov 01 06:18:45.648277 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.648231    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-credential-socket" pod "dfbc6deb-ab23-4706-b60e-2eb924fbb008" (UID: "dfbc6deb-ab23-4706-b60e-2eb924fbb008"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:18:45.648277 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.648246    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-workload-socket" pod "dfbc6deb-ab23-4706-b60e-2eb924fbb008" (UID: "dfbc6deb-ab23-4706-b60e-2eb924fbb008"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:18:45.648467 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.648374    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istiod-ca-cert" pod "dfbc6deb-ab23-4706-b60e-2eb924fbb008" (UID: "dfbc6deb-ab23-4706-b60e-2eb924fbb008"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:18:45.648467 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.648452    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:45.648544 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.648473    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:45.648544 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.648489    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:45.648544 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.648507    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:45.649748 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.649724    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-token" pod "dfbc6deb-ab23-4706-b60e-2eb924fbb008" (UID: "dfbc6deb-ab23-4706-b60e-2eb924fbb008"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:18:45.650233 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.650200    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-podinfo" pod "dfbc6deb-ab23-4706-b60e-2eb924fbb008" (UID: "dfbc6deb-ab23-4706-b60e-2eb924fbb008"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:18:45.650354 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.650248    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-envoy" pod "dfbc6deb-ab23-4706-b60e-2eb924fbb008" (UID: "dfbc6deb-ab23-4706-b60e-2eb924fbb008"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:18:45.650354 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.650274    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbc6deb-ab23-4706-b60e-2eb924fbb008-kube-api-access-bqbxp" pod "dfbc6deb-ab23-4706-b60e-2eb924fbb008" (UID: "dfbc6deb-ab23-4706-b60e-2eb924fbb008"). InnerVolumeSpecName "kube-api-access-bqbxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:18:45.749593 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.749562    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:45.749593 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.749591    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:45.749804 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.749605    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:45.749804 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.749614    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqbxp\" (UniqueName: \"kubernetes.io/projected/dfbc6deb-ab23-4706-b60e-2eb924fbb008-kube-api-access-bqbxp\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:45.749804 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.749624    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbc6deb-ab23-4706-b60e-2eb924fbb008-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:18:45.985417 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.985376    5619 generic.go:356] "Generic (PLEG): container finished" podID="dfbc6deb-ab23-4706-b60e-2eb924fbb008" containerID="c7f052316e9f6ca44265b3eb5ff0153e88ac7f51b5264fea791c0c13c6835ad1" exitCode=0
Nov 01 06:18:45.985604 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.985423    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"
Nov 01 06:18:45.985604 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.985445    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2" event={"ID":"dfbc6deb-ab23-4706-b60e-2eb924fbb008","Type":"ContainerDied","Data":"c7f052316e9f6ca44265b3eb5ff0153e88ac7f51b5264fea791c0c13c6835ad1"}
Nov 01 06:18:45.985604 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.985515    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2" event={"ID":"dfbc6deb-ab23-4706-b60e-2eb924fbb008","Type":"ContainerDied","Data":"93e31a67c5e6fc2c3175cfb22d451209fc7a44329fedf1568cbcbc44cf928de9"}
Nov 01 06:18:45.985604 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.985539    5619 scope.go:122] "RemoveContainer" containerID="c7f052316e9f6ca44265b3eb5ff0153e88ac7f51b5264fea791c0c13c6835ad1"
Nov 01 06:18:45.996297 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.996278    5619 scope.go:122] "RemoveContainer" containerID="c7f052316e9f6ca44265b3eb5ff0153e88ac7f51b5264fea791c0c13c6835ad1"
Nov 01 06:18:45.996649 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:18:45.996613    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f052316e9f6ca44265b3eb5ff0153e88ac7f51b5264fea791c0c13c6835ad1\": container with ID starting with c7f052316e9f6ca44265b3eb5ff0153e88ac7f51b5264fea791c0c13c6835ad1 not found: ID does not exist" containerID="c7f052316e9f6ca44265b3eb5ff0153e88ac7f51b5264fea791c0c13c6835ad1"
Nov 01 06:18:45.996649 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:45.996640    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f052316e9f6ca44265b3eb5ff0153e88ac7f51b5264fea791c0c13c6835ad1"} err="failed to get container status \"c7f052316e9f6ca44265b3eb5ff0153e88ac7f51b5264fea791c0c13c6835ad1\": rpc error: code = NotFound desc = could not find container \"c7f052316e9f6ca44265b3eb5ff0153e88ac7f51b5264fea791c0c13c6835ad1\": container with ID starting with c7f052316e9f6ca44265b3eb5ff0153e88ac7f51b5264fea791c0c13c6835ad1 not found: ID does not exist"
Nov 01 06:18:46.153476 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:46.153445    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"]
Nov 01 06:18:46.204055 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:46.204017    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-thxj2"]
Nov 01 06:18:46.963996 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:46.963948    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfbc6deb-ab23-4706-b60e-2eb924fbb008" path="/var/lib/kubelet/pods/dfbc6deb-ab23-4706-b60e-2eb924fbb008/volumes"
Nov 01 06:18:46.994388 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:46.994345    5619 generic.go:356] "Generic (PLEG): container finished" podID="1e3999df-3132-435d-93b4-0ed000180c4d" containerID="00bc0c5b50df72d72aadaa78e1a251246d6a940fb3e54678207cbc0422614d01" exitCode=1
Nov 01 06:18:46.994587 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:46.994428    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj" event={"ID":"1e3999df-3132-435d-93b4-0ed000180c4d","Type":"ContainerDied","Data":"00bc0c5b50df72d72aadaa78e1a251246d6a940fb3e54678207cbc0422614d01"}
Nov 01 06:18:46.994950 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:46.994932    5619 scope.go:122] "RemoveContainer" containerID="00bc0c5b50df72d72aadaa78e1a251246d6a940fb3e54678207cbc0422614d01"
Nov 01 06:18:48.005057 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:48.005020    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj" event={"ID":"1e3999df-3132-435d-93b4-0ed000180c4d","Type":"ContainerStarted","Data":"414ec24fa0b279e28804efe42b75f705899efc2850e86c2ac0863e5a8d9130d6"}
Nov 01 06:18:48.005499 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:48.005237    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"
Nov 01 06:18:48.700535 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:48.700503    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:53.508568 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:53.508528    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:54.016529 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:54.016498    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"
Nov 01 06:18:55.881655 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:18:55.881612    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:18:57.999405 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:57.999367    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"]
Nov 01 06:18:58.000111 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.000089    5619 cpu_manager.go:414] "containerMap: removing container" podUID="dfbc6deb-ab23-4706-b60e-2eb924fbb008" containerName="istio-proxy"
Nov 01 06:18:58.000212 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.000113    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfbc6deb-ab23-4706-b60e-2eb924fbb008" containerName="istio-proxy"
Nov 01 06:18:58.000292 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.000278    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfbc6deb-ab23-4706-b60e-2eb924fbb008" containerName="istio-proxy"
Nov 01 06:18:58.003699 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.003678    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"
Nov 01 06:18:58.058444 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.058419    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2a4cda2f-serving-cert\""
Nov 01 06:18:58.066200 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.066150    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"]
Nov 01 06:18:58.124865 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.124832    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx7dw\" (UniqueName: \"kubernetes.io/projected/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-kube-api-access-tx7dw\") pod \"maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc\" (UID: \"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"
Nov 01 06:18:58.125959 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.124883    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-maas-api-tls\") pod \"maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc\" (UID: \"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"
Nov 01 06:18:58.125959 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.125011    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-metrics-certs\") pod \"maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc\" (UID: \"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"
Nov 01 06:18:58.125959 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.125042    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-openshift-service-ca\") pod \"maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc\" (UID: \"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"
Nov 01 06:18:58.196923 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.196886    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:58.226005 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.225978    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx7dw\" (UniqueName: \"kubernetes.io/projected/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-kube-api-access-tx7dw\") pod \"maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc\" (UID: \"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"
Nov 01 06:18:58.226131 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.226021    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-maas-api-tls\") pod \"maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc\" (UID: \"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"
Nov 01 06:18:58.226131 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.226066    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-metrics-certs\") pod \"maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc\" (UID: \"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"
Nov 01 06:18:58.226131 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.226083    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-openshift-service-ca\") pod \"maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc\" (UID: \"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"
Nov 01 06:18:58.226241 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:18:58.226199    5619 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-serving-cert: secret "maas-api-e2e-ait-2a4cda2f-serving-cert" not found
Nov 01 06:18:58.226280 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:18:58.226271    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-maas-api-tls podName:52a81433-2eb5-4b1e-8a3a-ed2bb60409c2 nodeName:}" failed. No retries permitted until 2026-11-01 06:18:58.726253076 +0000 UTC m=+1975.937076578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-maas-api-tls") pod "maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc" (UID: "52a81433-2eb5-4b1e-8a3a-ed2bb60409c2") : secret "maas-api-e2e-ait-2a4cda2f-serving-cert" not found
Nov 01 06:18:58.226779 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.226762    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-openshift-service-ca\") pod \"maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc\" (UID: \"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"
Nov 01 06:18:58.228502 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.228483    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-metrics-certs\") pod \"maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc\" (UID: \"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"
Nov 01 06:18:58.288445 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.288410    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx7dw\" (UniqueName: \"kubernetes.io/projected/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-kube-api-access-tx7dw\") pod \"maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc\" (UID: \"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"
Nov 01 06:18:58.356955 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.356914    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:58.500789 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.500756    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:18:58.731698 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.731665    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-maas-api-tls\") pod \"maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc\" (UID: \"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"
Nov 01 06:18:58.734080 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.734055    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-maas-api-tls\") pod \"maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc\" (UID: \"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"
Nov 01 06:18:58.921994 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:58.921962    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"
Nov 01 06:18:59.087260 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:59.087178    5619 generic.go:356] "Generic (PLEG): container finished" podID="182e059b-08d3-4ea4-9a37-3bdc44581625" containerID="60da1d2de41805be3601e18106624ba7eea709026d5469985898569c8091de49" exitCode=0
Nov 01 06:18:59.087746 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:59.087253    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb" event={"ID":"182e059b-08d3-4ea4-9a37-3bdc44581625","Type":"ContainerDied","Data":"60da1d2de41805be3601e18106624ba7eea709026d5469985898569c8091de49"}
Nov 01 06:18:59.087814 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:59.087768    5619 scope.go:122] "RemoveContainer" containerID="60da1d2de41805be3601e18106624ba7eea709026d5469985898569c8091de49"
Nov 01 06:18:59.089216 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:59.089183    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:18:59.089312 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:59.089252    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:18:59.089312 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:59.089290    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:18:59.347555 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:18:59.347517    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a81433_2eb5_4b1e_8a3a_ed2bb60409c2.slice/crio-76ee71db872ab9c7700c5b78bb9e6d7db30dc2e08f3728c02e9325061505e423 WatchSource:0}: Error finding container 76ee71db872ab9c7700c5b78bb9e6d7db30dc2e08f3728c02e9325061505e423: Status 404 returned error can't find the container with id 76ee71db872ab9c7700c5b78bb9e6d7db30dc2e08f3728c02e9325061505e423
Nov 01 06:18:59.365483 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:59.365454    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"]
Nov 01 06:18:59.719380 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:18:59.719240    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:00.100727 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:00.100691    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb" event={"ID":"182e059b-08d3-4ea4-9a37-3bdc44581625","Type":"ContainerStarted","Data":"54e1ee34a189fd8676ef0e56cc458ef698f2f68e688776e17ccdfdb07fe39075"}
Nov 01 06:19:00.102369 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:00.102345    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc" event={"ID":"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2","Type":"ContainerStarted","Data":"76ee71db872ab9c7700c5b78bb9e6d7db30dc2e08f3728c02e9325061505e423"}
Nov 01 06:19:00.706580 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:00.706542    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:19:00.707531 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:00.707506    5619 patch_prober.go:29] interesting pod/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 06:19:00.707659 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:00.707556    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb" podUID="182e059b-08d3-4ea4-9a37-3bdc44581625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 06:19:01.116128 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:01.116077    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc" event={"ID":"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2","Type":"ContainerStarted","Data":"c3e2fee60c64ffabb67d56f201f0ec10a3a743a96a32ca3d26946b83f86b2dae"}
Nov 01 06:19:01.116656 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:01.116458    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"
Nov 01 06:19:01.237889 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:01.237831    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc" podStartSLOduration=3.473972369 podStartE2EDuration="4.237814364s" totalImagesPullingTime="763.841995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:18:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:01.237513637 +0000 UTC m=+1978.448337146" watchObservedRunningTime="2026-11-01 06:19:01.237814364 +0000 UTC m=+1978.448637872"
Nov 01 06:19:01.707481 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:01.707445    5619 patch_prober.go:29] interesting pod/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 06:19:01.707706 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:01.707501    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb" podUID="182e059b-08d3-4ea4-9a37-3bdc44581625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 06:19:02.568347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:02.564423    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:19:02.706906 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:02.706872    5619 patch_prober.go:29] interesting pod/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 06:19:02.707078 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:02.706933    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb" podUID="182e059b-08d3-4ea4-9a37-3bdc44581625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 06:19:03.260348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:03.258388    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:03.706600 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:03.706568    5619 patch_prober.go:29] interesting pod/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 06:19:03.706973 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:03.706627    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb" podUID="182e059b-08d3-4ea4-9a37-3bdc44581625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 06:19:04.707192 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:04.707156    5619 patch_prober.go:29] interesting pod/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 06:19:04.707622 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:04.707216    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb" podUID="182e059b-08d3-4ea4-9a37-3bdc44581625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 06:19:04.787980 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:04.787937    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:04.935840 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:04.935794    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:05.710692 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:05.710663    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:19:05.712005 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:05.711979    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:19:06.046001 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:06.045920    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:06.834247 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:06.834213    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"]
Nov 01 06:19:06.837597 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:06.837574    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"
Nov 01 06:19:06.890978 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:06.890948    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"agc-cred-d344d6f1c604\""
Nov 01 06:19:06.891131 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:06.891022    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"agc-cred-28460a5a0934\""
Nov 01 06:19:06.891206 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:06.891148    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"payload-processing-external-model-plugins-e2e-extmod-b-3986bf\""
Nov 01 06:19:06.891206 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:06.891155    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"kube-root-ca.crt\""
Nov 01 06:19:06.942153 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:06.942119    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"routing-overlay\""
Nov 01 06:19:06.942474 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:06.942450    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"payload-processing-external-model-e2e--cf3b8510-dockercfg-ftjt9\""
Nov 01 06:19:06.944764 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:06.944735    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"]
Nov 01 06:19:06.952633 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:06.952600    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjqwn\" (UniqueName: \"kubernetes.io/projected/79df3e10-85e6-4216-9505-4d67d326f7b5-kube-api-access-hjqwn\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r\" (UID: \"79df3e10-85e6-4216-9505-4d67d326f7b5\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"
Nov 01 06:19:06.952731 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:06.952715    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79df3e10-85e6-4216-9505-4d67d326f7b5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r\" (UID: \"79df3e10-85e6-4216-9505-4d67d326f7b5\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"
Nov 01 06:19:06.952813 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:06.952798    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79df3e10-85e6-4216-9505-4d67d326f7b5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r\" (UID: \"79df3e10-85e6-4216-9505-4d67d326f7b5\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"
Nov 01 06:19:06.952853 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:06.952836    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79df3e10-85e6-4216-9505-4d67d326f7b5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r\" (UID: \"79df3e10-85e6-4216-9505-4d67d326f7b5\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"
Nov 01 06:19:06.991098 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:06.991076    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"agc-cred-3feee30ca48b\""
Nov 01 06:19:06.991098 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:06.991085    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"openshift-service-ca.crt\""
Nov 01 06:19:06.991251 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:06.991080    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"agc-cred-c01d3031634c\""
Nov 01 06:19:07.053917 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:07.053889    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjqwn\" (UniqueName: \"kubernetes.io/projected/79df3e10-85e6-4216-9505-4d67d326f7b5-kube-api-access-hjqwn\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r\" (UID: \"79df3e10-85e6-4216-9505-4d67d326f7b5\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"
Nov 01 06:19:07.054059 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:07.054015    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79df3e10-85e6-4216-9505-4d67d326f7b5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r\" (UID: \"79df3e10-85e6-4216-9505-4d67d326f7b5\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"
Nov 01 06:19:07.054059 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:07.054057    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79df3e10-85e6-4216-9505-4d67d326f7b5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r\" (UID: \"79df3e10-85e6-4216-9505-4d67d326f7b5\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"
Nov 01 06:19:07.054199 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:07.054080    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79df3e10-85e6-4216-9505-4d67d326f7b5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r\" (UID: \"79df3e10-85e6-4216-9505-4d67d326f7b5\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"
Nov 01 06:19:07.054808 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:07.054784    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79df3e10-85e6-4216-9505-4d67d326f7b5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r\" (UID: \"79df3e10-85e6-4216-9505-4d67d326f7b5\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"
Nov 01 06:19:07.055021 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:07.054994    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79df3e10-85e6-4216-9505-4d67d326f7b5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r\" (UID: \"79df3e10-85e6-4216-9505-4d67d326f7b5\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"
Nov 01 06:19:07.056850 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:07.056830    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79df3e10-85e6-4216-9505-4d67d326f7b5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r\" (UID: \"79df3e10-85e6-4216-9505-4d67d326f7b5\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"
Nov 01 06:19:07.115583 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:07.115554    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjqwn\" (UniqueName: \"kubernetes.io/projected/79df3e10-85e6-4216-9505-4d67d326f7b5-kube-api-access-hjqwn\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r\" (UID: \"79df3e10-85e6-4216-9505-4d67d326f7b5\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"
Nov 01 06:19:07.131159 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:07.131138    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"
Nov 01 06:19:07.149778 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:07.149754    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"
Nov 01 06:19:07.532639 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:07.532605    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"]
Nov 01 06:19:07.534203 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:19:07.534177    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79df3e10_85e6_4216_9505_4d67d326f7b5.slice/crio-a4a1c6f6144a6f3f46220b665415a5718c6347fe6567eb9dd2377760b095d853 WatchSource:0}: Error finding container a4a1c6f6144a6f3f46220b665415a5718c6347fe6567eb9dd2377760b095d853: Status 404 returned error can't find the container with id a4a1c6f6144a6f3f46220b665415a5718c6347fe6567eb9dd2377760b095d853
Nov 01 06:19:08.064056 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:08.064010    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:08.163660 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:08.163625    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r" event={"ID":"79df3e10-85e6-4216-9505-4d67d326f7b5","Type":"ContainerStarted","Data":"07e33daab1132805e145100037022165bc29c46df924ca1ab9fbcf61f844449f"}
Nov 01 06:19:08.163857 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:08.163690    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r" event={"ID":"79df3e10-85e6-4216-9505-4d67d326f7b5","Type":"ContainerStarted","Data":"a4a1c6f6144a6f3f46220b665415a5718c6347fe6567eb9dd2377760b095d853"}
Nov 01 06:19:08.163857 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:08.163719    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"
Nov 01 06:19:08.281969 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:08.281917    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r" podStartSLOduration=2.281903958 podStartE2EDuration="2.281903958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:08.281434712 +0000 UTC m=+1985.492258220" watchObservedRunningTime="2026-11-01 06:19:08.281903958 +0000 UTC m=+1985.492727466"
Nov 01 06:19:11.771900 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:11.771816    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:11.922281 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:11.922240    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:12.070627 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:12.070524    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:12.785744 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:12.785708    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:17.534269 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:17.534220    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:17.821035 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:17.820949    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:17.972135 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:17.972098    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:18.332507 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:18.332462    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:18.816118 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:18.816078    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:19.030508 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:19.030475    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"]
Nov 01 06:19:19.034030 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:19.033998    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r" podUID="79df3e10-85e6-4216-9505-4d67d326f7b5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:19:19.080763 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:19.080678    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"]
Nov 01 06:19:19.084062 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:19.084040    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"
Nov 01 06:19:19.145914 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:19.145880    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"]
Nov 01 06:19:19.206497 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:19.206127    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17f135ad-e539-4d6b-8944-e28da2040064-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7\" (UID: \"17f135ad-e539-4d6b-8944-e28da2040064\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"
Nov 01 06:19:19.206497 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:19.206180    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f135ad-e539-4d6b-8944-e28da2040064-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7\" (UID: \"17f135ad-e539-4d6b-8944-e28da2040064\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"
Nov 01 06:19:19.206497 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:19.206208    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlq46\" (UniqueName: \"kubernetes.io/projected/17f135ad-e539-4d6b-8944-e28da2040064-kube-api-access-zlq46\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7\" (UID: \"17f135ad-e539-4d6b-8944-e28da2040064\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"
Nov 01 06:19:19.206497 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:19.206458    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17f135ad-e539-4d6b-8944-e28da2040064-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7\" (UID: \"17f135ad-e539-4d6b-8944-e28da2040064\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"
Nov 01 06:19:19.307657 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:19.307625    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17f135ad-e539-4d6b-8944-e28da2040064-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7\" (UID: \"17f135ad-e539-4d6b-8944-e28da2040064\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"
Nov 01 06:19:19.307861 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:19.307760    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17f135ad-e539-4d6b-8944-e28da2040064-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7\" (UID: \"17f135ad-e539-4d6b-8944-e28da2040064\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"
Nov 01 06:19:19.307861 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:19.307807    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f135ad-e539-4d6b-8944-e28da2040064-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7\" (UID: \"17f135ad-e539-4d6b-8944-e28da2040064\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"
Nov 01 06:19:19.307861 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:19.307855    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlq46\" (UniqueName: \"kubernetes.io/projected/17f135ad-e539-4d6b-8944-e28da2040064-kube-api-access-zlq46\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7\" (UID: \"17f135ad-e539-4d6b-8944-e28da2040064\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"
Nov 01 06:19:19.308490 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:19.308468    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17f135ad-e539-4d6b-8944-e28da2040064-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7\" (UID: \"17f135ad-e539-4d6b-8944-e28da2040064\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"
Nov 01 06:19:19.308588 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:19.308569    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f135ad-e539-4d6b-8944-e28da2040064-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7\" (UID: \"17f135ad-e539-4d6b-8944-e28da2040064\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"
Nov 01 06:19:19.310262 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:19.310234    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17f135ad-e539-4d6b-8944-e28da2040064-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7\" (UID: \"17f135ad-e539-4d6b-8944-e28da2040064\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"
Nov 01 06:19:19.375067 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:19.375028    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlq46\" (UniqueName: \"kubernetes.io/projected/17f135ad-e539-4d6b-8944-e28da2040064-kube-api-access-zlq46\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7\" (UID: \"17f135ad-e539-4d6b-8944-e28da2040064\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"
Nov 01 06:19:19.396964 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:19.396936    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"
Nov 01 06:19:19.577524 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:19.577462    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:19.748131 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:19.748038    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:19.792231 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:19.792196    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"]
Nov 01 06:19:19.792561 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:19:19.792537    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f135ad_e539_4d6b_8944_e28da2040064.slice/crio-c8c1a86c62eafc7d176fea8f92c6efa40bfd5191e96d850279b87fa1d264a4d7 WatchSource:0}: Error finding container c8c1a86c62eafc7d176fea8f92c6efa40bfd5191e96d850279b87fa1d264a4d7: Status 404 returned error can't find the container with id c8c1a86c62eafc7d176fea8f92c6efa40bfd5191e96d850279b87fa1d264a4d7
Nov 01 06:19:20.247447 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:20.247411    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7" event={"ID":"17f135ad-e539-4d6b-8944-e28da2040064","Type":"ContainerStarted","Data":"2202d5dcfcebfc4df2699493d1dd3699c3b9940d06e4a878fac7083ffb348ba8"}
Nov 01 06:19:20.247826 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:20.247494    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7" event={"ID":"17f135ad-e539-4d6b-8944-e28da2040064","Type":"ContainerStarted","Data":"c8c1a86c62eafc7d176fea8f92c6efa40bfd5191e96d850279b87fa1d264a4d7"}
Nov 01 06:19:20.247826 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:20.247519    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"
Nov 01 06:19:20.371344 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:20.368074    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7" podStartSLOduration=1.368052794 podStartE2EDuration="1.368052794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:20.365630787 +0000 UTC m=+1997.576454300" watchObservedRunningTime="2026-11-01 06:19:20.368052794 +0000 UTC m=+1997.578876304"
Nov 01 06:19:21.168348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:21.166876    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:21.379314 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:21.379278    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:22.406938 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:22.406891    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:22.558872 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:22.558835    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:22.785682 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:22.785591    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:24.031928 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:24.031887    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r" podUID="79df3e10-85e6-4216-9505-4d67d326f7b5" containerName="payload-processing" containerID="cri-o://07e33daab1132805e145100037022165bc29c46df924ca1ab9fbcf61f844449f" gracePeriod=25
Nov 01 06:19:24.305107 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:24.305026    5619 generic.go:356] "Generic (PLEG): container finished" podID="79df3e10-85e6-4216-9505-4d67d326f7b5" containerID="07e33daab1132805e145100037022165bc29c46df924ca1ab9fbcf61f844449f" exitCode=0
Nov 01 06:19:24.305265 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:24.305148    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r" event={"ID":"79df3e10-85e6-4216-9505-4d67d326f7b5","Type":"ContainerDied","Data":"07e33daab1132805e145100037022165bc29c46df924ca1ab9fbcf61f844449f"}
Nov 01 06:19:24.333426 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:24.333391    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"
Nov 01 06:19:24.464100 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:24.463884    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79df3e10-85e6-4216-9505-4d67d326f7b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79df3e10-85e6-4216-9505-4d67d326f7b5-plugins-config-volume\") pod \"79df3e10-85e6-4216-9505-4d67d326f7b5\" (UID: \"79df3e10-85e6-4216-9505-4d67d326f7b5\") "
Nov 01 06:19:24.464100 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:24.463957    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79df3e10-85e6-4216-9505-4d67d326f7b5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79df3e10-85e6-4216-9505-4d67d326f7b5-routing-overlay-volume\") pod \"79df3e10-85e6-4216-9505-4d67d326f7b5\" (UID: \"79df3e10-85e6-4216-9505-4d67d326f7b5\") "
Nov 01 06:19:24.464100 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:24.464014    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79df3e10-85e6-4216-9505-4d67d326f7b5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/79df3e10-85e6-4216-9505-4d67d326f7b5-provider-credentials\") pod \"79df3e10-85e6-4216-9505-4d67d326f7b5\" (UID: \"79df3e10-85e6-4216-9505-4d67d326f7b5\") "
Nov 01 06:19:24.464100 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:24.464054    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79df3e10-85e6-4216-9505-4d67d326f7b5-kube-api-access-hjqwn\" (UniqueName: \"kubernetes.io/projected/79df3e10-85e6-4216-9505-4d67d326f7b5-kube-api-access-hjqwn\") pod \"79df3e10-85e6-4216-9505-4d67d326f7b5\" (UID: \"79df3e10-85e6-4216-9505-4d67d326f7b5\") "
Nov 01 06:19:24.465812 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:24.465593    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79df3e10-85e6-4216-9505-4d67d326f7b5-routing-overlay-volume" pod "79df3e10-85e6-4216-9505-4d67d326f7b5" (UID: "79df3e10-85e6-4216-9505-4d67d326f7b5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:19:24.466467 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:24.466156    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79df3e10-85e6-4216-9505-4d67d326f7b5-plugins-config-volume" pod "79df3e10-85e6-4216-9505-4d67d326f7b5" (UID: "79df3e10-85e6-4216-9505-4d67d326f7b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:19:24.467459 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:24.467432    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79df3e10-85e6-4216-9505-4d67d326f7b5-kube-api-access-hjqwn" pod "79df3e10-85e6-4216-9505-4d67d326f7b5" (UID: "79df3e10-85e6-4216-9505-4d67d326f7b5"). InnerVolumeSpecName "kube-api-access-hjqwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:19:24.468888 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:24.468855    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79df3e10-85e6-4216-9505-4d67d326f7b5-provider-credentials" pod "79df3e10-85e6-4216-9505-4d67d326f7b5" (UID: "79df3e10-85e6-4216-9505-4d67d326f7b5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:19:24.521301 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:24.521266    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:24.566728 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:24.566657    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79df3e10-85e6-4216-9505-4d67d326f7b5-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:19:24.566728 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:24.566684    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjqwn\" (UniqueName: \"kubernetes.io/projected/79df3e10-85e6-4216-9505-4d67d326f7b5-kube-api-access-hjqwn\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:19:24.566728 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:24.566694    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79df3e10-85e6-4216-9505-4d67d326f7b5-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:19:24.566728 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:24.566707    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79df3e10-85e6-4216-9505-4d67d326f7b5-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:19:24.684508 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:24.684465    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:24.890059 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:24.890022    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:25.040390 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:25.040348    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:25.313375 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:25.313282    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"
Nov 01 06:19:25.313508 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:25.313280    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r" event={"ID":"79df3e10-85e6-4216-9505-4d67d326f7b5","Type":"ContainerDied","Data":"a4a1c6f6144a6f3f46220b665415a5718c6347fe6567eb9dd2377760b095d853"}
Nov 01 06:19:25.313508 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:25.313428    5619 scope.go:122] "RemoveContainer" containerID="07e33daab1132805e145100037022165bc29c46df924ca1ab9fbcf61f844449f"
Nov 01 06:19:25.480684 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:25.480646    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"]
Nov 01 06:19:25.531538 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:25.531507    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-b97994x9r"]
Nov 01 06:19:25.880310 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:19:25.880279    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:19:26.050170 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:26.050137    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:26.197835 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:26.197734    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:26.322316 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:26.322283    5619 generic.go:356] "Generic (PLEG): container finished" podID="93542be2-aa8a-4d1f-ade6-33dd834c5758" containerID="66bc50bf8e637149ede5b711b34b7c833d6e4393672d06bcd7c1b07c37406877" exitCode=0
Nov 01 06:19:26.322509 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:26.322360    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt" event={"ID":"93542be2-aa8a-4d1f-ade6-33dd834c5758","Type":"ContainerDied","Data":"66bc50bf8e637149ede5b711b34b7c833d6e4393672d06bcd7c1b07c37406877"}
Nov 01 06:19:26.322809 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:26.322793    5619 scope.go:122] "RemoveContainer" containerID="66bc50bf8e637149ede5b711b34b7c833d6e4393672d06bcd7c1b07c37406877"
Nov 01 06:19:26.324271 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:26.324242    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:19:26.324405 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:26.324356    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:19:26.324477 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:26.324415    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:19:26.962618 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:26.962582    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79df3e10-85e6-4216-9505-4d67d326f7b5" path="/var/lib/kubelet/pods/79df3e10-85e6-4216-9505-4d67d326f7b5/volumes"
Nov 01 06:19:27.259036 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:27.258952    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:27.331075 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:27.331041    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt" event={"ID":"93542be2-aa8a-4d1f-ade6-33dd834c5758","Type":"ContainerStarted","Data":"4363063ae8c674dde239f41bf33a9abaa58b11ea76e09d816dc52248f52310d7"}
Nov 01 06:19:27.584235 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:27.584132    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:28.298833 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:28.298790    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:19:28.299234 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:28.299078    5619 patch_prober.go:29] interesting pod/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 01 06:19:28.299234 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:28.299137    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt" podUID="93542be2-aa8a-4d1f-ade6-33dd834c5758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 01 06:19:29.299335 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:29.299291    5619 patch_prober.go:29] interesting pod/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 01 06:19:29.299731 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:29.299360    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt" podUID="93542be2-aa8a-4d1f-ade6-33dd834c5758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 01 06:19:30.298885 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.298841    5619 patch_prober.go:29] interesting pod/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 01 06:19:30.299075 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.298910    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt" podUID="93542be2-aa8a-4d1f-ade6-33dd834c5758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 01 06:19:30.309075 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.309040    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"]
Nov 01 06:19:30.312311 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.312275    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7" podUID="17f135ad-e539-4d6b-8944-e28da2040064" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:19:30.372347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.372011    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"]
Nov 01 06:19:30.373349 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.372955    5619 cpu_manager.go:414] "containerMap: removing container" podUID="79df3e10-85e6-4216-9505-4d67d326f7b5" containerName="payload-processing"
Nov 01 06:19:30.373349 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.372982    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79df3e10-85e6-4216-9505-4d67d326f7b5" containerName="payload-processing"
Nov 01 06:19:30.373349 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.373148    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="79df3e10-85e6-4216-9505-4d67d326f7b5" containerName="payload-processing"
Nov 01 06:19:30.380609 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.380585    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"
Nov 01 06:19:30.425452 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.425420    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"]
Nov 01 06:19:30.446561 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.446526    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjpzb\" (UniqueName: \"kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-kube-api-access-zjpzb\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4\" (UID: \"5ec32d57-998e-4646-885b-094f7fbaafd7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"
Nov 01 06:19:30.446723 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.446594    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5ec32d57-998e-4646-885b-094f7fbaafd7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4\" (UID: \"5ec32d57-998e-4646-885b-094f7fbaafd7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"
Nov 01 06:19:30.446771 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.446732    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4\" (UID: \"5ec32d57-998e-4646-885b-094f7fbaafd7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"
Nov 01 06:19:30.446771 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.446764    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec32d57-998e-4646-885b-094f7fbaafd7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4\" (UID: \"5ec32d57-998e-4646-885b-094f7fbaafd7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"
Nov 01 06:19:30.482541 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.482497    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3986bf\"/\"agc-cred-6204f08c208e\""
Nov 01 06:19:30.547675 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.547638    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5ec32d57-998e-4646-885b-094f7fbaafd7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4\" (UID: \"5ec32d57-998e-4646-885b-094f7fbaafd7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"
Nov 01 06:19:30.547863 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.547766    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4\" (UID: \"5ec32d57-998e-4646-885b-094f7fbaafd7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"
Nov 01 06:19:30.547863 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.547817    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec32d57-998e-4646-885b-094f7fbaafd7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4\" (UID: \"5ec32d57-998e-4646-885b-094f7fbaafd7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"
Nov 01 06:19:30.548064 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.547934    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjpzb\" (UniqueName: \"kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-kube-api-access-zjpzb\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4\" (UID: \"5ec32d57-998e-4646-885b-094f7fbaafd7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"
Nov 01 06:19:30.548471 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.548442    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5ec32d57-998e-4646-885b-094f7fbaafd7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4\" (UID: \"5ec32d57-998e-4646-885b-094f7fbaafd7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"
Nov 01 06:19:30.548780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.548757    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec32d57-998e-4646-885b-094f7fbaafd7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4\" (UID: \"5ec32d57-998e-4646-885b-094f7fbaafd7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"
Nov 01 06:19:30.585007 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.584947    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4\" (UID: \"5ec32d57-998e-4646-885b-094f7fbaafd7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"
Nov 01 06:19:30.610602 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.610570    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjpzb\" (UniqueName: \"kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-kube-api-access-zjpzb\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4\" (UID: \"5ec32d57-998e-4646-885b-094f7fbaafd7\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"
Nov 01 06:19:30.697301 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:30.697267    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"
Nov 01 06:19:31.102647 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:31.102616    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"]
Nov 01 06:19:31.103756 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:19:31.103722    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec32d57_998e_4646_885b_094f7fbaafd7.slice/crio-ba09403cb714ad8d3cae26085b6555fc045b0fbf6d01e837c3d7de1697e144fe WatchSource:0}: Error finding container ba09403cb714ad8d3cae26085b6555fc045b0fbf6d01e837c3d7de1697e144fe: Status 404 returned error can't find the container with id ba09403cb714ad8d3cae26085b6555fc045b0fbf6d01e837c3d7de1697e144fe
Nov 01 06:19:31.299026 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:31.298981    5619 patch_prober.go:29] interesting pod/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 01 06:19:31.299227 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:31.299036    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt" podUID="93542be2-aa8a-4d1f-ade6-33dd834c5758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 01 06:19:31.323975 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:31.323943    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:31.380634 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:31.380539    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4" event={"ID":"5ec32d57-998e-4646-885b-094f7fbaafd7","Type":"ContainerStarted","Data":"55b13ca88f8f899b45c35bcf8530483bbf330da8436588c7d2913b160b45f0c0"}
Nov 01 06:19:31.380634 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:31.380595    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4" event={"ID":"5ec32d57-998e-4646-885b-094f7fbaafd7","Type":"ContainerStarted","Data":"ba09403cb714ad8d3cae26085b6555fc045b0fbf6d01e837c3d7de1697e144fe"}
Nov 01 06:19:31.381256 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:31.381238    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"
Nov 01 06:19:31.505740 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:31.505675    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4" podStartSLOduration=1.505634171 podStartE2EDuration="1.505634171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:31.501789743 +0000 UTC m=+2008.712613253" watchObservedRunningTime="2026-11-01 06:19:31.505634171 +0000 UTC m=+2008.716457681"
Nov 01 06:19:32.298829 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:32.298793    5619 patch_prober.go:29] interesting pod/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 01 06:19:32.299030 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:32.298849    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt" podUID="93542be2-aa8a-4d1f-ade6-33dd834c5758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 01 06:19:32.299373 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:32.299343    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:33.303599 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:33.303565    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:19:33.304008 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:33.303928    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:19:33.304851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:33.304826    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:19:35.144922 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:35.144892    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:35.309581 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:35.309540    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7" podUID="17f135ad-e539-4d6b-8944-e28da2040064" containerName="payload-processing" containerID="cri-o://2202d5dcfcebfc4df2699493d1dd3699c3b9940d06e4a878fac7083ffb348ba8" gracePeriod=25
Nov 01 06:19:35.559987 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:35.559962    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"
Nov 01 06:19:35.643389 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:35.643362    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17f135ad-e539-4d6b-8944-e28da2040064-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17f135ad-e539-4d6b-8944-e28da2040064-routing-overlay-volume\") pod \"17f135ad-e539-4d6b-8944-e28da2040064\" (UID: \"17f135ad-e539-4d6b-8944-e28da2040064\") "
Nov 01 06:19:35.643555 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:35.643411    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f135ad-e539-4d6b-8944-e28da2040064-kube-api-access-zlq46\" (UniqueName: \"kubernetes.io/projected/17f135ad-e539-4d6b-8944-e28da2040064-kube-api-access-zlq46\") pod \"17f135ad-e539-4d6b-8944-e28da2040064\" (UID: \"17f135ad-e539-4d6b-8944-e28da2040064\") "
Nov 01 06:19:35.643555 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:35.643498    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17f135ad-e539-4d6b-8944-e28da2040064-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f135ad-e539-4d6b-8944-e28da2040064-plugins-config-volume\") pod \"17f135ad-e539-4d6b-8944-e28da2040064\" (UID: \"17f135ad-e539-4d6b-8944-e28da2040064\") "
Nov 01 06:19:35.643747 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:35.643676    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f135ad-e539-4d6b-8944-e28da2040064-provider-credentials\" (UniqueName: \"kubernetes.io/projected/17f135ad-e539-4d6b-8944-e28da2040064-provider-credentials\") pod \"17f135ad-e539-4d6b-8944-e28da2040064\" (UID: \"17f135ad-e539-4d6b-8944-e28da2040064\") "
Nov 01 06:19:35.643808 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:35.643756    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f135ad-e539-4d6b-8944-e28da2040064-routing-overlay-volume" pod "17f135ad-e539-4d6b-8944-e28da2040064" (UID: "17f135ad-e539-4d6b-8944-e28da2040064"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:19:35.643864 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:35.643844    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f135ad-e539-4d6b-8944-e28da2040064-plugins-config-volume" pod "17f135ad-e539-4d6b-8944-e28da2040064" (UID: "17f135ad-e539-4d6b-8944-e28da2040064"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:19:35.644174 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:35.644152    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17f135ad-e539-4d6b-8944-e28da2040064-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:19:35.644174 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:35.644174    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f135ad-e539-4d6b-8944-e28da2040064-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:19:35.645643 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:35.645617    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f135ad-e539-4d6b-8944-e28da2040064-provider-credentials" pod "17f135ad-e539-4d6b-8944-e28da2040064" (UID: "17f135ad-e539-4d6b-8944-e28da2040064"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:19:35.645706 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:35.645636    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f135ad-e539-4d6b-8944-e28da2040064-kube-api-access-zlq46" pod "17f135ad-e539-4d6b-8944-e28da2040064" (UID: "17f135ad-e539-4d6b-8944-e28da2040064"). InnerVolumeSpecName "kube-api-access-zlq46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:19:35.745583 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:35.745503    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlq46\" (UniqueName: \"kubernetes.io/projected/17f135ad-e539-4d6b-8944-e28da2040064-kube-api-access-zlq46\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:19:35.745583 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:35.745532    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17f135ad-e539-4d6b-8944-e28da2040064-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:19:36.418188 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:36.418156    5619 generic.go:356] "Generic (PLEG): container finished" podID="17f135ad-e539-4d6b-8944-e28da2040064" containerID="2202d5dcfcebfc4df2699493d1dd3699c3b9940d06e4a878fac7083ffb348ba8" exitCode=0
Nov 01 06:19:36.418691 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:36.418201    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"
Nov 01 06:19:36.418691 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:36.418244    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7" event={"ID":"17f135ad-e539-4d6b-8944-e28da2040064","Type":"ContainerDied","Data":"2202d5dcfcebfc4df2699493d1dd3699c3b9940d06e4a878fac7083ffb348ba8"}
Nov 01 06:19:36.418691 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:36.418308    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7" event={"ID":"17f135ad-e539-4d6b-8944-e28da2040064","Type":"ContainerDied","Data":"c8c1a86c62eafc7d176fea8f92c6efa40bfd5191e96d850279b87fa1d264a4d7"}
Nov 01 06:19:36.418691 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:36.418351    5619 scope.go:122] "RemoveContainer" containerID="2202d5dcfcebfc4df2699493d1dd3699c3b9940d06e4a878fac7083ffb348ba8"
Nov 01 06:19:36.428659 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:36.428644    5619 scope.go:122] "RemoveContainer" containerID="2202d5dcfcebfc4df2699493d1dd3699c3b9940d06e4a878fac7083ffb348ba8"
Nov 01 06:19:36.428909 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:19:36.428890    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2202d5dcfcebfc4df2699493d1dd3699c3b9940d06e4a878fac7083ffb348ba8\": container with ID starting with 2202d5dcfcebfc4df2699493d1dd3699c3b9940d06e4a878fac7083ffb348ba8 not found: ID does not exist" containerID="2202d5dcfcebfc4df2699493d1dd3699c3b9940d06e4a878fac7083ffb348ba8"
Nov 01 06:19:36.428975 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:36.428915    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2202d5dcfcebfc4df2699493d1dd3699c3b9940d06e4a878fac7083ffb348ba8"} err="failed to get container status \"2202d5dcfcebfc4df2699493d1dd3699c3b9940d06e4a878fac7083ffb348ba8\": rpc error: code = NotFound desc = could not find container \"2202d5dcfcebfc4df2699493d1dd3699c3b9940d06e4a878fac7083ffb348ba8\": container with ID starting with 2202d5dcfcebfc4df2699493d1dd3699c3b9940d06e4a878fac7083ffb348ba8 not found: ID does not exist"
Nov 01 06:19:36.595291 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:36.595259    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"]
Nov 01 06:19:36.607822 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:36.607796    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-5c77hsxn7"]
Nov 01 06:19:36.961634 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:36.961594    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17f135ad-e539-4d6b-8944-e28da2040064" path="/var/lib/kubelet/pods/17f135ad-e539-4d6b-8944-e28da2040064/volumes"
Nov 01 06:19:37.559370 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:37.559313    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:38.191849 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:38.191812    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:38.333833 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:38.333805    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:42.361465 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.361431    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"]
Nov 01 06:19:42.361999 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.361984    5619 cpu_manager.go:414] "containerMap: removing container" podUID="17f135ad-e539-4d6b-8944-e28da2040064" containerName="payload-processing"
Nov 01 06:19:42.362047 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.362000    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f135ad-e539-4d6b-8944-e28da2040064" containerName="payload-processing"
Nov 01 06:19:42.362096 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.362086    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f135ad-e539-4d6b-8944-e28da2040064" containerName="payload-processing"
Nov 01 06:19:42.366475 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.366456    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"
Nov 01 06:19:42.370809 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.370772    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:42.423506 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.423473    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"agc-cred-bdcae79313a2\""
Nov 01 06:19:42.426799 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.426767    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"]
Nov 01 06:19:42.427222 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.427190    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"agc-cred-b2178f695dad\""
Nov 01 06:19:42.474392 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.474352    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"agc-cred-af808cf4bef1\""
Nov 01 06:19:42.474623 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.474370    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"payload-processing-external-model-e2e--a835acb2-dockercfg-kqgqq\""
Nov 01 06:19:42.474759 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.474627    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"kube-root-ca.crt\""
Nov 01 06:19:42.474886 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.474721    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"openshift-service-ca.crt\""
Nov 01 06:19:42.474886 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.474767    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"routing-overlay\""
Nov 01 06:19:42.474886 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.474837    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7d25db\""
Nov 01 06:19:42.530723 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.530687    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc5vm\" (UniqueName: \"kubernetes.io/projected/35b34d83-9725-475a-90b8-66955f34f5a6-kube-api-access-bc5vm\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b\" (UID: \"35b34d83-9725-475a-90b8-66955f34f5a6\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"
Nov 01 06:19:42.530865 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.530760    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35b34d83-9725-475a-90b8-66955f34f5a6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b\" (UID: \"35b34d83-9725-475a-90b8-66955f34f5a6\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"
Nov 01 06:19:42.530917 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.530861    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35b34d83-9725-475a-90b8-66955f34f5a6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b\" (UID: \"35b34d83-9725-475a-90b8-66955f34f5a6\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"
Nov 01 06:19:42.531028 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.531012    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b34d83-9725-475a-90b8-66955f34f5a6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b\" (UID: \"35b34d83-9725-475a-90b8-66955f34f5a6\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"
Nov 01 06:19:42.632374 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.632282    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc5vm\" (UniqueName: \"kubernetes.io/projected/35b34d83-9725-475a-90b8-66955f34f5a6-kube-api-access-bc5vm\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b\" (UID: \"35b34d83-9725-475a-90b8-66955f34f5a6\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"
Nov 01 06:19:42.632374 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.632356    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35b34d83-9725-475a-90b8-66955f34f5a6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b\" (UID: \"35b34d83-9725-475a-90b8-66955f34f5a6\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"
Nov 01 06:19:42.632570 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.632383    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35b34d83-9725-475a-90b8-66955f34f5a6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b\" (UID: \"35b34d83-9725-475a-90b8-66955f34f5a6\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"
Nov 01 06:19:42.632570 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.632449    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b34d83-9725-475a-90b8-66955f34f5a6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b\" (UID: \"35b34d83-9725-475a-90b8-66955f34f5a6\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"
Nov 01 06:19:42.633130 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.633104    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35b34d83-9725-475a-90b8-66955f34f5a6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b\" (UID: \"35b34d83-9725-475a-90b8-66955f34f5a6\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"
Nov 01 06:19:42.633221 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.633202    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b34d83-9725-475a-90b8-66955f34f5a6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b\" (UID: \"35b34d83-9725-475a-90b8-66955f34f5a6\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"
Nov 01 06:19:42.635012 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.634988    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35b34d83-9725-475a-90b8-66955f34f5a6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b\" (UID: \"35b34d83-9725-475a-90b8-66955f34f5a6\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"
Nov 01 06:19:42.694608 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.694575    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc5vm\" (UniqueName: \"kubernetes.io/projected/35b34d83-9725-475a-90b8-66955f34f5a6-kube-api-access-bc5vm\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b\" (UID: \"35b34d83-9725-475a-90b8-66955f34f5a6\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"
Nov 01 06:19:42.980760 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:42.980672    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"
Nov 01 06:19:43.355695 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:43.355668    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"]
Nov 01 06:19:43.358495 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:19:43.358458    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b34d83_9725_475a_90b8_66955f34f5a6.slice/crio-51b061c05e0b25ff40b84ee17f531c801d468d53cc6d6c6cff6a0997a02cac79 WatchSource:0}: Error finding container 51b061c05e0b25ff40b84ee17f531c801d468d53cc6d6c6cff6a0997a02cac79: Status 404 returned error can't find the container with id 51b061c05e0b25ff40b84ee17f531c801d468d53cc6d6c6cff6a0997a02cac79
Nov 01 06:19:43.400599 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:43.400577    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"
Nov 01 06:19:43.487280 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:43.487240    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b" event={"ID":"35b34d83-9725-475a-90b8-66955f34f5a6","Type":"ContainerStarted","Data":"0813a05ae40685f917f5027922f87f7f28efe249308fa755585b48b2e2898697"}
Nov 01 06:19:43.487458 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:43.487314    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"
Nov 01 06:19:43.487458 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:43.487370    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b" event={"ID":"35b34d83-9725-475a-90b8-66955f34f5a6","Type":"ContainerStarted","Data":"51b061c05e0b25ff40b84ee17f531c801d468d53cc6d6c6cff6a0997a02cac79"}
Nov 01 06:19:44.890527 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:44.890476    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b" podStartSLOduration=2.89046071 podStartE2EDuration="2.89046071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:43.642285533 +0000 UTC m=+2020.853109042" watchObservedRunningTime="2026-11-01 06:19:44.89046071 +0000 UTC m=+2022.101284255"
Nov 01 06:19:44.892820 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:44.892794    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"]
Nov 01 06:19:44.896888 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:44.896865    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"
Nov 01 06:19:45.002951 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.002920    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"agc-cred-123858cde139\""
Nov 01 06:19:45.003172 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.002936    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"agc-cred-c481d4f49e7f\""
Nov 01 06:19:45.003258 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.002950    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"routing-overlay\""
Nov 01 06:19:45.003258 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.002951    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8f45a2\""
Nov 01 06:19:45.003258 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.002978    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"agc-cred-d6d3484085b2\""
Nov 01 06:19:45.003258 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.002995    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"kube-root-ca.crt\""
Nov 01 06:19:45.003258 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.002993    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"openshift-service-ca.crt\""
Nov 01 06:19:45.003462 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.003045    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"payload-processing-external-model-e2e--cb1d2723-dockercfg-smk8q\""
Nov 01 06:19:45.005596 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.005574    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"]
Nov 01 06:19:45.061943 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.061914    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7\" (UID: \"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"
Nov 01 06:19:45.062316 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.062290    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7\" (UID: \"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"
Nov 01 06:19:45.062389 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.062347    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d56h\" (UniqueName: \"kubernetes.io/projected/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-kube-api-access-5d56h\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7\" (UID: \"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"
Nov 01 06:19:45.062554 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.062509    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7\" (UID: \"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"
Nov 01 06:19:45.163563 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.163494    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7\" (UID: \"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"
Nov 01 06:19:45.163677 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.163603    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7\" (UID: \"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"
Nov 01 06:19:45.163677 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.163621    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d56h\" (UniqueName: \"kubernetes.io/projected/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-kube-api-access-5d56h\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7\" (UID: \"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"
Nov 01 06:19:45.163754 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.163676    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7\" (UID: \"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"
Nov 01 06:19:45.164248 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.164224    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7\" (UID: \"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"
Nov 01 06:19:45.164345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.164301    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7\" (UID: \"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"
Nov 01 06:19:45.166092 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.166065    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7\" (UID: \"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"
Nov 01 06:19:45.225483 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.225459    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d56h\" (UniqueName: \"kubernetes.io/projected/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-kube-api-access-5d56h\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7\" (UID: \"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"
Nov 01 06:19:45.511733 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.511652    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"
Nov 01 06:19:45.886470 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:45.886428    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"]
Nov 01 06:19:45.887615 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:19:45.887590    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3307b4e_28b5_4a3c_ac2c_ddf988561ecf.slice/crio-284e50f2f2ba778f05dbe5ce97a002a44f8bd71df23a4d95a8f4a25d4a06ea38 WatchSource:0}: Error finding container 284e50f2f2ba778f05dbe5ce97a002a44f8bd71df23a4d95a8f4a25d4a06ea38: Status 404 returned error can't find the container with id 284e50f2f2ba778f05dbe5ce97a002a44f8bd71df23a4d95a8f4a25d4a06ea38
Nov 01 06:19:46.507055 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:46.507014    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7" event={"ID":"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf","Type":"ContainerStarted","Data":"d92acc608a18c807ea9fe53af12653d92600935f33ad7db412ef9400d9163d90"}
Nov 01 06:19:46.507516 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:46.507195    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"
Nov 01 06:19:46.507516 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:46.507245    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7" event={"ID":"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf","Type":"ContainerStarted","Data":"284e50f2f2ba778f05dbe5ce97a002a44f8bd71df23a4d95a8f4a25d4a06ea38"}
Nov 01 06:19:46.625734 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:46.625679    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7" podStartSLOduration=2.62566173 podStartE2EDuration="2.62566173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:46.625242851 +0000 UTC m=+2023.836066361" watchObservedRunningTime="2026-11-01 06:19:46.62566173 +0000 UTC m=+2023.836485240"
Nov 01 06:19:47.143718 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:47.143683    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:49.793836 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:49.793797    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:50.058613 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:50.058518    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:50.215197 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:50.215153    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:50.367565 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:50.367523    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:51.963851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:51.963819    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:54.496554 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:54.496523    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"
Nov 01 06:19:55.880019 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:19:55.879981    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:19:56.190435 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.190346    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:56.485912 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.485832    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"]
Nov 01 06:19:56.489888 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.489867    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.497194 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.497166    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.497318 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.497263    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.497439 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.497374    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.497439 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.497399    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.497584 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.497562    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.497659 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.497643    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.497708 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.497693    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.497809 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.497784    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg6dx\" (UniqueName: \"kubernetes.io/projected/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-kube-api-access-jg6dx\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.497874 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.497852    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.598712 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.598676    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.598900 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.598744    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.598900 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.598791    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.598900 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.598827    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg6dx\" (UniqueName: \"kubernetes.io/projected/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-kube-api-access-jg6dx\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.598900 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.598854    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.599125 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.598937    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.599125 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.598991    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.599125 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.599067    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.599125 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.599099    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.599125 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.599122    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.599431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.599405    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.599884 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.599854    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.600004 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.599900    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.600101 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.600077    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.601753 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.601721    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.601865 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.601795    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.662789 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.662760    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.662956 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.662931    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg6dx\" (UniqueName: \"kubernetes.io/projected/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-kube-api-access-jg6dx\") pod \"maas-default-gateway-openshift-default-855f9574cf-8c4ww\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:56.694908 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.694865    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"]
Nov 01 06:19:56.786466 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.786361    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:19:56.805301 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:56.805277    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:57.240964 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:57.240935    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"]
Nov 01 06:19:57.242352 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:19:57.242304    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7114a9_2bb2_44ec_aa8d_11a9b7cc0cea.slice/crio-79e96a96af2de3956d16c91442ebb2e401bf62c11f80ab05dd13e9d2371df474 WatchSource:0}: Error finding container 79e96a96af2de3956d16c91442ebb2e401bf62c11f80ab05dd13e9d2371df474: Status 404 returned error can't find the container with id 79e96a96af2de3956d16c91442ebb2e401bf62c11f80ab05dd13e9d2371df474
Nov 01 06:19:57.245371 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:57.244864    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:19:57.245371 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:57.244934    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:19:57.245371 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:57.244979    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:19:57.516561 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:57.516460    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"
Nov 01 06:19:57.577090 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:57.577042    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww" event={"ID":"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea","Type":"ContainerStarted","Data":"e27fbac9595f220c71eab7004111fe0888f62db8ea53897b9b151dc0ed498cc3"}
Nov 01 06:19:57.577294 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:57.577108    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww" event={"ID":"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea","Type":"ContainerStarted","Data":"79e96a96af2de3956d16c91442ebb2e401bf62c11f80ab05dd13e9d2371df474"}
Nov 01 06:19:57.757404 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:57.757300    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww" podStartSLOduration=1.757281554 podStartE2EDuration="1.757281554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:57.756404917 +0000 UTC m=+2034.967228426" watchObservedRunningTime="2026-11-01 06:19:57.757281554 +0000 UTC m=+2034.968105063"
Nov 01 06:19:57.805948 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:57.805866    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:19:58.807786 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:58.807742    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8c4ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 01 06:19:58.808293 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:58.807817    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww" podUID="bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 01 06:19:59.805558 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:59.805524    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8c4ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 01 06:19:59.805749 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:19:59.805576    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww" podUID="bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 01 06:20:00.805854 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:00.805821    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8c4ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 01 06:20:00.806410 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:00.805885    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww" podUID="bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 01 06:20:01.806445 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:01.806411    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8c4ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 01 06:20:01.806825 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:01.806471    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww" podUID="bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 01 06:20:01.940838 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:01.940803    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:02.223359 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:02.219996    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:02.807382 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:02.805904    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8c4ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 01 06:20:02.807382 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:02.805981    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww" podUID="bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 01 06:20:03.805906 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:03.805816    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8c4ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 01 06:20:03.805906 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:03.805889    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww" podUID="bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 01 06:20:04.806179 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:04.806142    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8c4ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 01 06:20:04.806730 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:04.806239    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww" podUID="bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 01 06:20:06.094608 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:06.094562    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:20:06.095297 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:06.095265    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:20:06.096540 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:06.096514    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:20:07.509697 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:07.509580    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:11.608104 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.608071    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"]
Nov 01 06:20:11.613253 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.613236    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.712690 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.712650    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5tng\" (UniqueName: \"kubernetes.io/projected/69cd086b-554c-49c9-a471-831d40610ceb-kube-api-access-g5tng\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.712857 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.712706    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.712857 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.712837    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.712936 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.712901    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.713007 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.712989    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.713042 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.713021    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69cd086b-554c-49c9-a471-831d40610ceb-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.713106 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.713094    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69cd086b-554c-49c9-a471-831d40610ceb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.713141 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.713131    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.713185 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.713148    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69cd086b-554c-49c9-a471-831d40610ceb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.813967 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.813934    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5tng\" (UniqueName: \"kubernetes.io/projected/69cd086b-554c-49c9-a471-831d40610ceb-kube-api-access-g5tng\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.813967 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.813973    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.814183 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.813994    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.814183 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.814017    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.814183 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.814084    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.814183 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.814121    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69cd086b-554c-49c9-a471-831d40610ceb-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.814183 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.814165    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69cd086b-554c-49c9-a471-831d40610ceb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.814464 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.814215    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.814464 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.814243    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69cd086b-554c-49c9-a471-831d40610ceb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.814464 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.814395    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.814639 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.814459    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.814639 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.814499    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.814639 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.814621    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.814893 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.814874    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69cd086b-554c-49c9-a471-831d40610ceb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.816308 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.816283    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.816643 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.816624    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69cd086b-554c-49c9-a471-831d40610ceb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.828149 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.828122    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"]
Nov 01 06:20:11.876086 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.876007    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69cd086b-554c-49c9-a471-831d40610ceb-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.876086 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.876015    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5tng\" (UniqueName: \"kubernetes.io/projected/69cd086b-554c-49c9-a471-831d40610ceb-kube-api-access-g5tng\") pod \"maas-default-gateway-openshift-default-855f9574cf-76s2k\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:11.927056 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:11.927025    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:12.356902 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:12.356870    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"]
Nov 01 06:20:12.357615 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:20:12.357585    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69cd086b_554c_49c9_a471_831d40610ceb.slice/crio-68ee04acb249ee0a21c51b58706098df7df234407a311421227a3198a7b1002a WatchSource:0}: Error finding container 68ee04acb249ee0a21c51b58706098df7df234407a311421227a3198a7b1002a: Status 404 returned error can't find the container with id 68ee04acb249ee0a21c51b58706098df7df234407a311421227a3198a7b1002a
Nov 01 06:20:12.359762 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:12.359734    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:12.360447 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:12.360414    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:20:12.360555 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:12.360484    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:20:12.360555 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:12.360514    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:20:12.687936 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:12.687845    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k" event={"ID":"69cd086b-554c-49c9-a471-831d40610ceb","Type":"ContainerStarted","Data":"a135903a60c7cfdd5993c3726afbf831edd114bcd689f1f19b7971081968b924"}
Nov 01 06:20:12.687936 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:12.687909    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k" event={"ID":"69cd086b-554c-49c9-a471-831d40610ceb","Type":"ContainerStarted","Data":"68ee04acb249ee0a21c51b58706098df7df234407a311421227a3198a7b1002a"}
Nov 01 06:20:12.812432 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:12.812382    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k" podStartSLOduration=1.81236868 podStartE2EDuration="1.81236868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:20:12.809486389 +0000 UTC m=+2050.020309909" watchObservedRunningTime="2026-11-01 06:20:12.81236868 +0000 UTC m=+2050.023192195"
Nov 01 06:20:12.928042 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:12.927956    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:13.929442 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:13.929403    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-76s2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 01 06:20:13.929870 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:13.929466    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k" podUID="69cd086b-554c-49c9-a471-831d40610ceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 01 06:20:14.927431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:14.927395    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-76s2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 01 06:20:14.927615 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:14.927450    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k" podUID="69cd086b-554c-49c9-a471-831d40610ceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 01 06:20:15.928124 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:15.928085    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-76s2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 01 06:20:15.928590 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:15.928154    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k" podUID="69cd086b-554c-49c9-a471-831d40610ceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 01 06:20:16.928193 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:16.928155    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-76s2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 01 06:20:16.928680 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:16.928217    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k" podUID="69cd086b-554c-49c9-a471-831d40610ceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 01 06:20:17.124469 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:17.124433    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:17.928092 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:17.928051    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-76s2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 01 06:20:17.928295 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:17.928129    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k" podUID="69cd086b-554c-49c9-a471-831d40610ceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 01 06:20:18.842802 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:18.842762    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v"]
Nov 01 06:20:18.927599 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:18.927569    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-76s2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 01 06:20:18.927785 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:18.927622    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k" podUID="69cd086b-554c-49c9-a471-831d40610ceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 01 06:20:18.953909 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:18.953874    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2"]
Nov 01 06:20:19.202285 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:19.202200    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:19.347302 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:19.347267    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:19.928134 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:19.928099    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-76s2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 01 06:20:19.928308 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:19.928165    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k" podUID="69cd086b-554c-49c9-a471-831d40610ceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 01 06:20:20.953361 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:20.953300    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:20.954023 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:20.954000    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:20.964200 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:20.964174    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:20:21.228847 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:21.228779    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"]
Nov 01 06:20:21.955948 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:21.955916    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:23.128370 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:23.128314    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:23.843645 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:23.843606    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v" podUID="4eba5cde-05e2-4926-89c6-11231b9060ea" containerName="payload-processing" containerID="cri-o://ac85663bcc83222502a52312e9f8ed92e0a328271da8c9af77cc283e54eaa119" gracePeriod=25
Nov 01 06:20:23.954267 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:23.954223    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2" podUID="ffe3528c-3bc2-447c-93e5-ce82909e5fb1" containerName="payload-pre-processing" containerID="cri-o://5ba127f81be4b1c0fdd0af7dbcce149f7909504fb921be11b40e4a11c93838c4" gracePeriod=25
Nov 01 06:20:24.114638 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.114564    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v"
Nov 01 06:20:24.168500 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.168465    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"]
Nov 01 06:20:24.193275 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.193236    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eba5cde-05e2-4926-89c6-11231b9060ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eba5cde-05e2-4926-89c6-11231b9060ea-plugins-config-volume\") pod \"4eba5cde-05e2-4926-89c6-11231b9060ea\" (UID: \"4eba5cde-05e2-4926-89c6-11231b9060ea\") "
Nov 01 06:20:24.193461 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.193287    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eba5cde-05e2-4926-89c6-11231b9060ea-kube-api-access-whbqd\" (UniqueName: \"kubernetes.io/projected/4eba5cde-05e2-4926-89c6-11231b9060ea-kube-api-access-whbqd\") pod \"4eba5cde-05e2-4926-89c6-11231b9060ea\" (UID: \"4eba5cde-05e2-4926-89c6-11231b9060ea\") "
Nov 01 06:20:24.193740 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.193712    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eba5cde-05e2-4926-89c6-11231b9060ea-plugins-config-volume" pod "4eba5cde-05e2-4926-89c6-11231b9060ea" (UID: "4eba5cde-05e2-4926-89c6-11231b9060ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:20:24.193989 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.193968    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eba5cde-05e2-4926-89c6-11231b9060ea-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:24.195996 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.195965    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eba5cde-05e2-4926-89c6-11231b9060ea-kube-api-access-whbqd" pod "4eba5cde-05e2-4926-89c6-11231b9060ea" (UID: "4eba5cde-05e2-4926-89c6-11231b9060ea"). InnerVolumeSpecName "kube-api-access-whbqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:20:24.215483 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.215463    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2"
Nov 01 06:20:24.294587 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.294551    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe3528c-3bc2-447c-93e5-ce82909e5fb1-kube-api-access-9xlls\" (UniqueName: \"kubernetes.io/projected/ffe3528c-3bc2-447c-93e5-ce82909e5fb1-kube-api-access-9xlls\") pod \"ffe3528c-3bc2-447c-93e5-ce82909e5fb1\" (UID: \"ffe3528c-3bc2-447c-93e5-ce82909e5fb1\") "
Nov 01 06:20:24.294781 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.294705    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffe3528c-3bc2-447c-93e5-ce82909e5fb1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe3528c-3bc2-447c-93e5-ce82909e5fb1-plugins-config-volume\") pod \"ffe3528c-3bc2-447c-93e5-ce82909e5fb1\" (UID: \"ffe3528c-3bc2-447c-93e5-ce82909e5fb1\") "
Nov 01 06:20:24.295242 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.295216    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whbqd\" (UniqueName: \"kubernetes.io/projected/4eba5cde-05e2-4926-89c6-11231b9060ea-kube-api-access-whbqd\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:24.295643 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.295620    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe3528c-3bc2-447c-93e5-ce82909e5fb1-plugins-config-volume" pod "ffe3528c-3bc2-447c-93e5-ce82909e5fb1" (UID: "ffe3528c-3bc2-447c-93e5-ce82909e5fb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:20:24.297755 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.297732    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe3528c-3bc2-447c-93e5-ce82909e5fb1-kube-api-access-9xlls" pod "ffe3528c-3bc2-447c-93e5-ce82909e5fb1" (UID: "ffe3528c-3bc2-447c-93e5-ce82909e5fb1"). InnerVolumeSpecName "kube-api-access-9xlls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:20:24.395990 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.395922    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xlls\" (UniqueName: \"kubernetes.io/projected/ffe3528c-3bc2-447c-93e5-ce82909e5fb1-kube-api-access-9xlls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:24.395990 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.395949    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe3528c-3bc2-447c-93e5-ce82909e5fb1-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:24.768961 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.768885    5619 generic.go:356] "Generic (PLEG): container finished" podID="ffe3528c-3bc2-447c-93e5-ce82909e5fb1" containerID="5ba127f81be4b1c0fdd0af7dbcce149f7909504fb921be11b40e4a11c93838c4" exitCode=0
Nov 01 06:20:24.768961 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.768930    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2"
Nov 01 06:20:24.769137 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.768963    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2" event={"ID":"ffe3528c-3bc2-447c-93e5-ce82909e5fb1","Type":"ContainerDied","Data":"5ba127f81be4b1c0fdd0af7dbcce149f7909504fb921be11b40e4a11c93838c4"}
Nov 01 06:20:24.769137 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.769023    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2" event={"ID":"ffe3528c-3bc2-447c-93e5-ce82909e5fb1","Type":"ContainerDied","Data":"3033647ceb5fe6fd432d9449033481d59947446308d6f2a876bd0cebc59e2b93"}
Nov 01 06:20:24.769137 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.769048    5619 scope.go:122] "RemoveContainer" containerID="5ba127f81be4b1c0fdd0af7dbcce149f7909504fb921be11b40e4a11c93838c4"
Nov 01 06:20:24.770659 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.770633    5619 generic.go:356] "Generic (PLEG): container finished" podID="4eba5cde-05e2-4926-89c6-11231b9060ea" containerID="ac85663bcc83222502a52312e9f8ed92e0a328271da8c9af77cc283e54eaa119" exitCode=0
Nov 01 06:20:24.770796 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.770667    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v"
Nov 01 06:20:24.770796 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.770673    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v" event={"ID":"4eba5cde-05e2-4926-89c6-11231b9060ea","Type":"ContainerDied","Data":"ac85663bcc83222502a52312e9f8ed92e0a328271da8c9af77cc283e54eaa119"}
Nov 01 06:20:24.770796 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.770708    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v" event={"ID":"4eba5cde-05e2-4926-89c6-11231b9060ea","Type":"ContainerDied","Data":"48a307d52e5be665bb62346378c12cb6b58a5078d7607da8d3e6a3d0c6abfba3"}
Nov 01 06:20:24.780740 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.780724    5619 scope.go:122] "RemoveContainer" containerID="5ba127f81be4b1c0fdd0af7dbcce149f7909504fb921be11b40e4a11c93838c4"
Nov 01 06:20:24.781010 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:20:24.780986    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ba127f81be4b1c0fdd0af7dbcce149f7909504fb921be11b40e4a11c93838c4\": container with ID starting with 5ba127f81be4b1c0fdd0af7dbcce149f7909504fb921be11b40e4a11c93838c4 not found: ID does not exist" containerID="5ba127f81be4b1c0fdd0af7dbcce149f7909504fb921be11b40e4a11c93838c4"
Nov 01 06:20:24.781087 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.781020    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ba127f81be4b1c0fdd0af7dbcce149f7909504fb921be11b40e4a11c93838c4"} err="failed to get container status \"5ba127f81be4b1c0fdd0af7dbcce149f7909504fb921be11b40e4a11c93838c4\": rpc error: code = NotFound desc = could not find container \"5ba127f81be4b1c0fdd0af7dbcce149f7909504fb921be11b40e4a11c93838c4\": container with ID starting with 5ba127f81be4b1c0fdd0af7dbcce149f7909504fb921be11b40e4a11c93838c4 not found: ID does not exist"
Nov 01 06:20:24.781087 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.781045    5619 scope.go:122] "RemoveContainer" containerID="ac85663bcc83222502a52312e9f8ed92e0a328271da8c9af77cc283e54eaa119"
Nov 01 06:20:24.790240 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.790219    5619 scope.go:122] "RemoveContainer" containerID="ac85663bcc83222502a52312e9f8ed92e0a328271da8c9af77cc283e54eaa119"
Nov 01 06:20:24.790587 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:20:24.790559    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac85663bcc83222502a52312e9f8ed92e0a328271da8c9af77cc283e54eaa119\": container with ID starting with ac85663bcc83222502a52312e9f8ed92e0a328271da8c9af77cc283e54eaa119 not found: ID does not exist" containerID="ac85663bcc83222502a52312e9f8ed92e0a328271da8c9af77cc283e54eaa119"
Nov 01 06:20:24.790670 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.790598    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac85663bcc83222502a52312e9f8ed92e0a328271da8c9af77cc283e54eaa119"} err="failed to get container status \"ac85663bcc83222502a52312e9f8ed92e0a328271da8c9af77cc283e54eaa119\": rpc error: code = NotFound desc = could not find container \"ac85663bcc83222502a52312e9f8ed92e0a328271da8c9af77cc283e54eaa119\": container with ID starting with ac85663bcc83222502a52312e9f8ed92e0a328271da8c9af77cc283e54eaa119 not found: ID does not exist"
Nov 01 06:20:24.935951 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.935917    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2"]
Nov 01 06:20:24.986863 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:24.986833    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2a4cda2f-67b956c475-xg6x2"]
Nov 01 06:20:25.050834 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:25.050763    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v"]
Nov 01 06:20:25.056996 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:25.056971    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2a4cda2f-fb5f879f7-kzv2v"]
Nov 01 06:20:25.880841 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:20:25.880801    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:20:26.229438 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.229349    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7" podUID="e3307b4e-28b5-4a3c-ac2c-ddf988561ecf" containerName="payload-processing" containerID="cri-o://d92acc608a18c807ea9fe53af12653d92600935f33ad7db412ef9400d9163d90" gracePeriod=25
Nov 01 06:20:26.483782 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.483734    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"
Nov 01 06:20:26.621209 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.621175    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-provider-credentials\") pod \"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf\" (UID: \"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf\") "
Nov 01 06:20:26.621433 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.621259    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-plugins-config-volume\") pod \"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf\" (UID: \"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf\") "
Nov 01 06:20:26.621433 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.621334    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-routing-overlay-volume\") pod \"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf\" (UID: \"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf\") "
Nov 01 06:20:26.621433 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.621388    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-kube-api-access-5d56h\" (UniqueName: \"kubernetes.io/projected/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-kube-api-access-5d56h\") pod \"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf\" (UID: \"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf\") "
Nov 01 06:20:26.621756 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.621726    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-routing-overlay-volume" pod "e3307b4e-28b5-4a3c-ac2c-ddf988561ecf" (UID: "e3307b4e-28b5-4a3c-ac2c-ddf988561ecf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:20:26.621756 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.621738    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-plugins-config-volume" pod "e3307b4e-28b5-4a3c-ac2c-ddf988561ecf" (UID: "e3307b4e-28b5-4a3c-ac2c-ddf988561ecf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:20:26.622015 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.621994    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:26.622085 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.622022    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:26.623696 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.623669    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-provider-credentials" pod "e3307b4e-28b5-4a3c-ac2c-ddf988561ecf" (UID: "e3307b4e-28b5-4a3c-ac2c-ddf988561ecf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:20:26.623785 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.623745    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-kube-api-access-5d56h" pod "e3307b4e-28b5-4a3c-ac2c-ddf988561ecf" (UID: "e3307b4e-28b5-4a3c-ac2c-ddf988561ecf"). InnerVolumeSpecName "kube-api-access-5d56h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:20:26.723112 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.723084    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:26.723112 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.723112    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d56h\" (UniqueName: \"kubernetes.io/projected/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf-kube-api-access-5d56h\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:26.724703 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.724665    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:26.788584 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.788268    5619 generic.go:356] "Generic (PLEG): container finished" podID="e3307b4e-28b5-4a3c-ac2c-ddf988561ecf" containerID="d92acc608a18c807ea9fe53af12653d92600935f33ad7db412ef9400d9163d90" exitCode=0
Nov 01 06:20:26.788584 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.788504    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7" event={"ID":"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf","Type":"ContainerDied","Data":"d92acc608a18c807ea9fe53af12653d92600935f33ad7db412ef9400d9163d90"}
Nov 01 06:20:26.788584 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.788564    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7" event={"ID":"e3307b4e-28b5-4a3c-ac2c-ddf988561ecf","Type":"ContainerDied","Data":"284e50f2f2ba778f05dbe5ce97a002a44f8bd71df23a4d95a8f4a25d4a06ea38"}
Nov 01 06:20:26.788851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.788596    5619 scope.go:122] "RemoveContainer" containerID="d92acc608a18c807ea9fe53af12653d92600935f33ad7db412ef9400d9163d90"
Nov 01 06:20:26.788851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.788721    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"
Nov 01 06:20:26.802415 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.802392    5619 scope.go:122] "RemoveContainer" containerID="d92acc608a18c807ea9fe53af12653d92600935f33ad7db412ef9400d9163d90"
Nov 01 06:20:26.802723 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:20:26.802698    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d92acc608a18c807ea9fe53af12653d92600935f33ad7db412ef9400d9163d90\": container with ID starting with d92acc608a18c807ea9fe53af12653d92600935f33ad7db412ef9400d9163d90 not found: ID does not exist" containerID="d92acc608a18c807ea9fe53af12653d92600935f33ad7db412ef9400d9163d90"
Nov 01 06:20:26.802814 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.802737    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d92acc608a18c807ea9fe53af12653d92600935f33ad7db412ef9400d9163d90"} err="failed to get container status \"d92acc608a18c807ea9fe53af12653d92600935f33ad7db412ef9400d9163d90\": rpc error: code = NotFound desc = could not find container \"d92acc608a18c807ea9fe53af12653d92600935f33ad7db412ef9400d9163d90\": container with ID starting with d92acc608a18c807ea9fe53af12653d92600935f33ad7db412ef9400d9163d90 not found: ID does not exist"
Nov 01 06:20:26.962465 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.962431    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eba5cde-05e2-4926-89c6-11231b9060ea" path="/var/lib/kubelet/pods/4eba5cde-05e2-4926-89c6-11231b9060ea/volumes"
Nov 01 06:20:26.962848 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.962834    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffe3528c-3bc2-447c-93e5-ce82909e5fb1" path="/var/lib/kubelet/pods/ffe3528c-3bc2-447c-93e5-ce82909e5fb1/volumes"
Nov 01 06:20:26.963657 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.963634    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"]
Nov 01 06:20:26.970242 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:26.970216    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-55b9p7gf7"]
Nov 01 06:20:27.934030 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:27.933977    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:28.259299 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:28.259213    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:28.403049 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:28.403013    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:28.963334 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:28.963281    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3307b4e-28b5-4a3c-ac2c-ddf988561ecf" path="/var/lib/kubelet/pods/e3307b4e-28b5-4a3c-ac2c-ddf988561ecf/volumes"
Nov 01 06:20:29.168996 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:29.168957    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b" podUID="35b34d83-9725-475a-90b8-66955f34f5a6" containerName="payload-processing" containerID="cri-o://0813a05ae40685f917f5027922f87f7f28efe249308fa755585b48b2e2898697" gracePeriod=25
Nov 01 06:20:29.448392 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:29.448364    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"
Nov 01 06:20:29.556469 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:29.556394    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35b34d83-9725-475a-90b8-66955f34f5a6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35b34d83-9725-475a-90b8-66955f34f5a6-routing-overlay-volume\") pod \"35b34d83-9725-475a-90b8-66955f34f5a6\" (UID: \"35b34d83-9725-475a-90b8-66955f34f5a6\") "
Nov 01 06:20:29.556469 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:29.556450    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b34d83-9725-475a-90b8-66955f34f5a6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/35b34d83-9725-475a-90b8-66955f34f5a6-provider-credentials\") pod \"35b34d83-9725-475a-90b8-66955f34f5a6\" (UID: \"35b34d83-9725-475a-90b8-66955f34f5a6\") "
Nov 01 06:20:29.556648 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:29.556597    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35b34d83-9725-475a-90b8-66955f34f5a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b34d83-9725-475a-90b8-66955f34f5a6-plugins-config-volume\") pod \"35b34d83-9725-475a-90b8-66955f34f5a6\" (UID: \"35b34d83-9725-475a-90b8-66955f34f5a6\") "
Nov 01 06:20:29.556753 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:29.556730    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b34d83-9725-475a-90b8-66955f34f5a6-kube-api-access-bc5vm\" (UniqueName: \"kubernetes.io/projected/35b34d83-9725-475a-90b8-66955f34f5a6-kube-api-access-bc5vm\") pod \"35b34d83-9725-475a-90b8-66955f34f5a6\" (UID: \"35b34d83-9725-475a-90b8-66955f34f5a6\") "
Nov 01 06:20:29.556904 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:29.556780    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b34d83-9725-475a-90b8-66955f34f5a6-routing-overlay-volume" pod "35b34d83-9725-475a-90b8-66955f34f5a6" (UID: "35b34d83-9725-475a-90b8-66955f34f5a6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:20:29.557057 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:29.557031    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b34d83-9725-475a-90b8-66955f34f5a6-plugins-config-volume" pod "35b34d83-9725-475a-90b8-66955f34f5a6" (UID: "35b34d83-9725-475a-90b8-66955f34f5a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:20:29.557180 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:29.557164    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b34d83-9725-475a-90b8-66955f34f5a6-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:29.557224 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:29.557201    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35b34d83-9725-475a-90b8-66955f34f5a6-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:29.558960 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:29.558935    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b34d83-9725-475a-90b8-66955f34f5a6-kube-api-access-bc5vm" pod "35b34d83-9725-475a-90b8-66955f34f5a6" (UID: "35b34d83-9725-475a-90b8-66955f34f5a6"). InnerVolumeSpecName "kube-api-access-bc5vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:20:29.559060 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:29.559041    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b34d83-9725-475a-90b8-66955f34f5a6-provider-credentials" pod "35b34d83-9725-475a-90b8-66955f34f5a6" (UID: "35b34d83-9725-475a-90b8-66955f34f5a6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:20:29.658492 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:29.658452    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc5vm\" (UniqueName: \"kubernetes.io/projected/35b34d83-9725-475a-90b8-66955f34f5a6-kube-api-access-bc5vm\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:29.658492 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:29.658493    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35b34d83-9725-475a-90b8-66955f34f5a6-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:29.811107 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:29.811013    5619 generic.go:356] "Generic (PLEG): container finished" podID="35b34d83-9725-475a-90b8-66955f34f5a6" containerID="0813a05ae40685f917f5027922f87f7f28efe249308fa755585b48b2e2898697" exitCode=0
Nov 01 06:20:29.811107 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:29.811072    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"
Nov 01 06:20:29.811355 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:29.811077    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b" event={"ID":"35b34d83-9725-475a-90b8-66955f34f5a6","Type":"ContainerDied","Data":"0813a05ae40685f917f5027922f87f7f28efe249308fa755585b48b2e2898697"}
Nov 01 06:20:29.811355 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:29.811201    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b" event={"ID":"35b34d83-9725-475a-90b8-66955f34f5a6","Type":"ContainerDied","Data":"51b061c05e0b25ff40b84ee17f531c801d468d53cc6d6c6cff6a0997a02cac79"}
Nov 01 06:20:29.811355 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:29.811227    5619 scope.go:122] "RemoveContainer" containerID="0813a05ae40685f917f5027922f87f7f28efe249308fa755585b48b2e2898697"
Nov 01 06:20:29.821710 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:29.821690    5619 scope.go:122] "RemoveContainer" containerID="0813a05ae40685f917f5027922f87f7f28efe249308fa755585b48b2e2898697"
Nov 01 06:20:29.821996 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:20:29.821979    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0813a05ae40685f917f5027922f87f7f28efe249308fa755585b48b2e2898697\": container with ID starting with 0813a05ae40685f917f5027922f87f7f28efe249308fa755585b48b2e2898697 not found: ID does not exist" containerID="0813a05ae40685f917f5027922f87f7f28efe249308fa755585b48b2e2898697"
Nov 01 06:20:29.822043 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:29.822003    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0813a05ae40685f917f5027922f87f7f28efe249308fa755585b48b2e2898697"} err="failed to get container status \"0813a05ae40685f917f5027922f87f7f28efe249308fa755585b48b2e2898697\": rpc error: code = NotFound desc = could not find container \"0813a05ae40685f917f5027922f87f7f28efe249308fa755585b48b2e2898697\": container with ID starting with 0813a05ae40685f917f5027922f87f7f28efe249308fa755585b48b2e2898697 not found: ID does not exist"
Nov 01 06:20:29.978294 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:29.978262    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"]
Nov 01 06:20:30.029275 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:30.029245    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-b7fdb4s2b"]
Nov 01 06:20:30.964511 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:30.964473    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35b34d83-9725-475a-90b8-66955f34f5a6" path="/var/lib/kubelet/pods/35b34d83-9725-475a-90b8-66955f34f5a6/volumes"
Nov 01 06:20:31.554966 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:31.554928    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:36.332632 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:36.332590    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:41.068476 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:41.068390    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:41.898504 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:41.898466    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:42.054046 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:42.054002    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:45.080647 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.080609    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"]
Nov 01 06:20:45.081040 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.080828    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc" podUID="52a81433-2eb5-4b1e-8a3a-ed2bb60409c2" containerName="maas-api" containerID="cri-o://c3e2fee60c64ffabb67d56f201f0ec10a3a743a96a32ca3d26946b83f86b2dae" gracePeriod=30
Nov 01 06:20:45.187208 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.187171    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:45.349116 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.349087    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"
Nov 01 06:20:45.456877 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.456837    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-openshift-service-ca\") pod \"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2\" (UID: \"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2\") "
Nov 01 06:20:45.457065 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.456884    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-metrics-certs\") pod \"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2\" (UID: \"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2\") "
Nov 01 06:20:45.457187 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.457146    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-maas-api-tls\") pod \"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2\" (UID: \"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2\") "
Nov 01 06:20:45.457375 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.457317    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-kube-api-access-tx7dw\" (UniqueName: \"kubernetes.io/projected/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-kube-api-access-tx7dw\") pod \"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2\" (UID: \"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2\") "
Nov 01 06:20:45.457503 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.457425    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-openshift-service-ca" pod "52a81433-2eb5-4b1e-8a3a-ed2bb60409c2" (UID: "52a81433-2eb5-4b1e-8a3a-ed2bb60409c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:20:45.475375 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.457857    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:45.481446 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.480823    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-maas-api-tls" pod "52a81433-2eb5-4b1e-8a3a-ed2bb60409c2" (UID: "52a81433-2eb5-4b1e-8a3a-ed2bb60409c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:20:45.481446 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.480912    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-metrics-certs" pod "52a81433-2eb5-4b1e-8a3a-ed2bb60409c2" (UID: "52a81433-2eb5-4b1e-8a3a-ed2bb60409c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:20:45.481446 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.481175    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-kube-api-access-tx7dw" pod "52a81433-2eb5-4b1e-8a3a-ed2bb60409c2" (UID: "52a81433-2eb5-4b1e-8a3a-ed2bb60409c2"). InnerVolumeSpecName "kube-api-access-tx7dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:20:45.558897 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.558859    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:45.558897 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.558892    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx7dw\" (UniqueName: \"kubernetes.io/projected/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-kube-api-access-tx7dw\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:45.559093 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.558907    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:45.908736 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.908698    5619 generic.go:356] "Generic (PLEG): container finished" podID="52a81433-2eb5-4b1e-8a3a-ed2bb60409c2" containerID="c3e2fee60c64ffabb67d56f201f0ec10a3a743a96a32ca3d26946b83f86b2dae" exitCode=0
Nov 01 06:20:45.908992 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.908751    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"
Nov 01 06:20:45.908992 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.908789    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc" event={"ID":"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2","Type":"ContainerDied","Data":"c3e2fee60c64ffabb67d56f201f0ec10a3a743a96a32ca3d26946b83f86b2dae"}
Nov 01 06:20:45.908992 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.908855    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc" event={"ID":"52a81433-2eb5-4b1e-8a3a-ed2bb60409c2","Type":"ContainerDied","Data":"76ee71db872ab9c7700c5b78bb9e6d7db30dc2e08f3728c02e9325061505e423"}
Nov 01 06:20:45.908992 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.908888    5619 scope.go:122] "RemoveContainer" containerID="c3e2fee60c64ffabb67d56f201f0ec10a3a743a96a32ca3d26946b83f86b2dae"
Nov 01 06:20:45.919246 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.919227    5619 scope.go:122] "RemoveContainer" containerID="c3e2fee60c64ffabb67d56f201f0ec10a3a743a96a32ca3d26946b83f86b2dae"
Nov 01 06:20:45.919638 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:20:45.919600    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e2fee60c64ffabb67d56f201f0ec10a3a743a96a32ca3d26946b83f86b2dae\": container with ID starting with c3e2fee60c64ffabb67d56f201f0ec10a3a743a96a32ca3d26946b83f86b2dae not found: ID does not exist" containerID="c3e2fee60c64ffabb67d56f201f0ec10a3a743a96a32ca3d26946b83f86b2dae"
Nov 01 06:20:45.919738 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:45.919644    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e2fee60c64ffabb67d56f201f0ec10a3a743a96a32ca3d26946b83f86b2dae"} err="failed to get container status \"c3e2fee60c64ffabb67d56f201f0ec10a3a743a96a32ca3d26946b83f86b2dae\": rpc error: code = NotFound desc = could not find container \"c3e2fee60c64ffabb67d56f201f0ec10a3a743a96a32ca3d26946b83f86b2dae\": container with ID starting with c3e2fee60c64ffabb67d56f201f0ec10a3a743a96a32ca3d26946b83f86b2dae not found: ID does not exist"
Nov 01 06:20:46.082689 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:46.082644    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"]
Nov 01 06:20:46.086789 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:46.086762    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2a4cda2f-86cb759c58-kr8wc"]
Nov 01 06:20:46.682592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:46.682558    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:46.959528 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:46.959448    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:46.963602 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:46.963576    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52a81433-2eb5-4b1e-8a3a-ed2bb60409c2" path="/var/lib/kubelet/pods/52a81433-2eb5-4b1e-8a3a-ed2bb60409c2/volumes"
Nov 01 06:20:47.159952 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:47.159918    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:47.413148 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:47.413117    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"]
Nov 01 06:20:47.413359 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:47.413333    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt" podUID="93542be2-aa8a-4d1f-ade6-33dd834c5758" containerName="istio-proxy" containerID="cri-o://4363063ae8c674dde239f41bf33a9abaa58b11ea76e09d816dc52248f52310d7" gracePeriod=30
Nov 01 06:20:47.421484 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:47.421454    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:47.658929 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:47.658889    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:48.056106 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:48.056073    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:48.247339 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:48.247286    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:48.306729 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:48.306656    5619 patch_prober.go:29] interesting pod/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:20:48.306729 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:48.306710    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt" podUID="93542be2-aa8a-4d1f-ade6-33dd834c5758" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:20:48.483752 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:48.483715    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:48.936491 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:48.936456    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"]
Nov 01 06:20:48.937047 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:48.937026    5619 cpu_manager.go:414] "containerMap: removing container" podUID="ffe3528c-3bc2-447c-93e5-ce82909e5fb1" containerName="payload-pre-processing"
Nov 01 06:20:48.937149 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:48.937050    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe3528c-3bc2-447c-93e5-ce82909e5fb1" containerName="payload-pre-processing"
Nov 01 06:20:48.937149 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:48.937066    5619 cpu_manager.go:414] "containerMap: removing container" podUID="52a81433-2eb5-4b1e-8a3a-ed2bb60409c2" containerName="maas-api"
Nov 01 06:20:48.937149 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:48.937073    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a81433-2eb5-4b1e-8a3a-ed2bb60409c2" containerName="maas-api"
Nov 01 06:20:48.937149 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:48.937099    5619 cpu_manager.go:414] "containerMap: removing container" podUID="4eba5cde-05e2-4926-89c6-11231b9060ea" containerName="payload-processing"
Nov 01 06:20:48.937149 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:48.937107    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eba5cde-05e2-4926-89c6-11231b9060ea" containerName="payload-processing"
Nov 01 06:20:48.937149 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:48.937144    5619 cpu_manager.go:414] "containerMap: removing container" podUID="e3307b4e-28b5-4a3c-ac2c-ddf988561ecf" containerName="payload-processing"
Nov 01 06:20:48.937480 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:48.937151    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3307b4e-28b5-4a3c-ac2c-ddf988561ecf" containerName="payload-processing"
Nov 01 06:20:48.937480 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:48.937172    5619 cpu_manager.go:414] "containerMap: removing container" podUID="35b34d83-9725-475a-90b8-66955f34f5a6" containerName="payload-processing"
Nov 01 06:20:48.937480 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:48.937180    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35b34d83-9725-475a-90b8-66955f34f5a6" containerName="payload-processing"
Nov 01 06:20:48.937480 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:48.937344    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a81433-2eb5-4b1e-8a3a-ed2bb60409c2" containerName="maas-api"
Nov 01 06:20:48.937480 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:48.937358    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe3528c-3bc2-447c-93e5-ce82909e5fb1" containerName="payload-pre-processing"
Nov 01 06:20:48.937480 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:48.937372    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3307b4e-28b5-4a3c-ac2c-ddf988561ecf" containerName="payload-processing"
Nov 01 06:20:48.937480 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:48.937383    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eba5cde-05e2-4926-89c6-11231b9060ea" containerName="payload-processing"
Nov 01 06:20:48.937480 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:48.937392    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="35b34d83-9725-475a-90b8-66955f34f5a6" containerName="payload-processing"
Nov 01 06:20:48.942404 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:48.942382    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.045347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.041859    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e5d7100d-gw-openshift-default-dockercfg-whwgt\""
Nov 01 06:20:49.051343 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.050860    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:49.056265 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.056224    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-workload-certs\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.056455 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.056284    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb80df79-1ef0-4be1-8607-0394242a387a-istiod-ca-cert\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.056455 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.056335    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb80df79-1ef0-4be1-8607-0394242a387a-istio-token\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.057072 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.056970    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-credential-socket\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.057072 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.057030    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk4jm\" (UniqueName: \"kubernetes.io/projected/cb80df79-1ef0-4be1-8607-0394242a387a-kube-api-access-vk4jm\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.057072 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.057066    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-workload-socket\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.057241 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.057094    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-istio-envoy\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.057241 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.057170    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb80df79-1ef0-4be1-8607-0394242a387a-istio-podinfo\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.057358 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.057309    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-istio-data\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.107839 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.107800    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"]
Nov 01 06:20:49.158541 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.158501    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-istio-data\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.158757 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.158569    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-workload-certs\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.158757 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.158596    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb80df79-1ef0-4be1-8607-0394242a387a-istiod-ca-cert\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.158757 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.158629    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb80df79-1ef0-4be1-8607-0394242a387a-istio-token\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.158757 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.158713    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-credential-socket\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.158757 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.158752    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk4jm\" (UniqueName: \"kubernetes.io/projected/cb80df79-1ef0-4be1-8607-0394242a387a-kube-api-access-vk4jm\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.159036 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.158782    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-workload-socket\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.159036 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.158812    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-istio-envoy\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.159036 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.158861    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb80df79-1ef0-4be1-8607-0394242a387a-istio-podinfo\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.159036 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.158927    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-istio-data\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.159036 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.158941    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-workload-certs\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.159300 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.159248    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-workload-socket\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.159402 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.159386    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb80df79-1ef0-4be1-8607-0394242a387a-istiod-ca-cert\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.159500 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.159477    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-credential-socket\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.161111 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.161081    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-istio-envoy\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.161503 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.161488    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb80df79-1ef0-4be1-8607-0394242a387a-istio-podinfo\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.221200 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.221139    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk4jm\" (UniqueName: \"kubernetes.io/projected/cb80df79-1ef0-4be1-8607-0394242a387a-kube-api-access-vk4jm\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.221200 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.221147    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb80df79-1ef0-4be1-8607-0394242a387a-istio-token\") pod \"e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") " pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.256290 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.256252    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:49.269726 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.269685    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:49.698703 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.698677    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"]
Nov 01 06:20:49.700297 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:20:49.700269    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb80df79_1ef0_4be1_8607_0394242a387a.slice/crio-54f59daeb4ad6833ef5f0f1e760e0e08856b11f017edc42f3f5ee1d2cc9de644 WatchSource:0}: Error finding container 54f59daeb4ad6833ef5f0f1e760e0e08856b11f017edc42f3f5ee1d2cc9de644: Status 404 returned error can't find the container with id 54f59daeb4ad6833ef5f0f1e760e0e08856b11f017edc42f3f5ee1d2cc9de644
Nov 01 06:20:49.703524 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.703485    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:20:49.703597 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.703576    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:20:49.703635 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.703618    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:20:49.937641 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.937604    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2" event={"ID":"cb80df79-1ef0-4be1-8607-0394242a387a","Type":"ContainerStarted","Data":"b198e622e33cb9aa30a4246239a00504ab2f3c26cc77339c73a3b94e60995126"}
Nov 01 06:20:49.937845 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:49.937675    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2" event={"ID":"cb80df79-1ef0-4be1-8607-0394242a387a","Type":"ContainerStarted","Data":"54f59daeb4ad6833ef5f0f1e760e0e08856b11f017edc42f3f5ee1d2cc9de644"}
Nov 01 06:20:50.059726 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:50.059609    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2" podStartSLOduration=2.059589612 podStartE2EDuration="2.059589612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:20:50.058747917 +0000 UTC m=+2087.269571427" watchObservedRunningTime="2026-11-01 06:20:50.059589612 +0000 UTC m=+2087.270413123"
Nov 01 06:20:50.257205 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:50.257162    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:50.263669 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:50.263635    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:50.943358 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:50.943306    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:50.944345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:50.944288    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:20:52.197425 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.197392    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:52.453481 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.453397    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:52.672281 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.672235    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:52.690767 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.690744    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:20:52.810368 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.810272    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93542be2-aa8a-4d1f-ade6-33dd834c5758-kube-api-access-vt4hr\" (UniqueName: \"kubernetes.io/projected/93542be2-aa8a-4d1f-ade6-33dd834c5758-kube-api-access-vt4hr\") pod \"93542be2-aa8a-4d1f-ade6-33dd834c5758\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") "
Nov 01 06:20:52.810368 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.810317    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-workload-certs\") pod \"93542be2-aa8a-4d1f-ade6-33dd834c5758\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") "
Nov 01 06:20:52.810604 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.810427    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-data\") pod \"93542be2-aa8a-4d1f-ade6-33dd834c5758\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") "
Nov 01 06:20:52.810604 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.810477    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-credential-socket\") pod \"93542be2-aa8a-4d1f-ade6-33dd834c5758\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") "
Nov 01 06:20:52.810604 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.810514    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-podinfo\") pod \"93542be2-aa8a-4d1f-ade6-33dd834c5758\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") "
Nov 01 06:20:52.810604 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.810562    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93542be2-aa8a-4d1f-ade6-33dd834c5758-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93542be2-aa8a-4d1f-ade6-33dd834c5758-istiod-ca-cert\") pod \"93542be2-aa8a-4d1f-ade6-33dd834c5758\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") "
Nov 01 06:20:52.810813 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.810643    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-workload-socket\") pod \"93542be2-aa8a-4d1f-ade6-33dd834c5758\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") "
Nov 01 06:20:52.810813 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.810647    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-workload-certs" pod "93542be2-aa8a-4d1f-ade6-33dd834c5758" (UID: "93542be2-aa8a-4d1f-ade6-33dd834c5758"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:20:52.810813 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.810673    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-envoy\") pod \"93542be2-aa8a-4d1f-ade6-33dd834c5758\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") "
Nov 01 06:20:52.810813 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.810708    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-data" pod "93542be2-aa8a-4d1f-ade6-33dd834c5758" (UID: "93542be2-aa8a-4d1f-ade6-33dd834c5758"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:20:52.810813 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.810770    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-credential-socket" pod "93542be2-aa8a-4d1f-ade6-33dd834c5758" (UID: "93542be2-aa8a-4d1f-ade6-33dd834c5758"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:20:52.811037 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.810870    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-token\" (UniqueName: \"kubernetes.io/projected/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-token\") pod \"93542be2-aa8a-4d1f-ade6-33dd834c5758\" (UID: \"93542be2-aa8a-4d1f-ade6-33dd834c5758\") "
Nov 01 06:20:52.811037 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.810894    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-workload-socket" pod "93542be2-aa8a-4d1f-ade6-33dd834c5758" (UID: "93542be2-aa8a-4d1f-ade6-33dd834c5758"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:20:52.811037 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.810970    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93542be2-aa8a-4d1f-ade6-33dd834c5758-istiod-ca-cert" pod "93542be2-aa8a-4d1f-ade6-33dd834c5758" (UID: "93542be2-aa8a-4d1f-ade6-33dd834c5758"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:20:52.811367 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.811347    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:52.811468 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.811372    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:52.811468 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.811389    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93542be2-aa8a-4d1f-ade6-33dd834c5758-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:52.811468 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.811404    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:52.811468 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.811421    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:52.812944 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.812917    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-podinfo" pod "93542be2-aa8a-4d1f-ade6-33dd834c5758" (UID: "93542be2-aa8a-4d1f-ade6-33dd834c5758"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:20:52.813103 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.813081    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-envoy" pod "93542be2-aa8a-4d1f-ade6-33dd834c5758" (UID: "93542be2-aa8a-4d1f-ade6-33dd834c5758"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:20:52.813204 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.813188    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-token" pod "93542be2-aa8a-4d1f-ade6-33dd834c5758" (UID: "93542be2-aa8a-4d1f-ade6-33dd834c5758"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:20:52.813258 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.813215    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93542be2-aa8a-4d1f-ade6-33dd834c5758-kube-api-access-vt4hr" pod "93542be2-aa8a-4d1f-ade6-33dd834c5758" (UID: "93542be2-aa8a-4d1f-ade6-33dd834c5758"). InnerVolumeSpecName "kube-api-access-vt4hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:20:52.912154 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.912113    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:52.912154 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.912147    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:52.912154 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.912159    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93542be2-aa8a-4d1f-ade6-33dd834c5758-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:52.912447 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.912168    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt4hr\" (UniqueName: \"kubernetes.io/projected/93542be2-aa8a-4d1f-ade6-33dd834c5758-kube-api-access-vt4hr\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:20:52.957377 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.956298    5619 generic.go:356] "Generic (PLEG): container finished" podID="93542be2-aa8a-4d1f-ade6-33dd834c5758" containerID="4363063ae8c674dde239f41bf33a9abaa58b11ea76e09d816dc52248f52310d7" exitCode=0
Nov 01 06:20:52.958067 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.958037    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"
Nov 01 06:20:52.970295 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.967002    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt" event={"ID":"93542be2-aa8a-4d1f-ade6-33dd834c5758","Type":"ContainerDied","Data":"4363063ae8c674dde239f41bf33a9abaa58b11ea76e09d816dc52248f52310d7"}
Nov 01 06:20:52.970295 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.967066    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt" event={"ID":"93542be2-aa8a-4d1f-ade6-33dd834c5758","Type":"ContainerDied","Data":"f94d5402fcd8b2a1daa107890c052bd0628afea8446617225dc7763fb22faca1"}
Nov 01 06:20:52.970295 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.967111    5619 scope.go:122] "RemoveContainer" containerID="4363063ae8c674dde239f41bf33a9abaa58b11ea76e09d816dc52248f52310d7"
Nov 01 06:20:52.986363 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.986343    5619 scope.go:122] "RemoveContainer" containerID="66bc50bf8e637149ede5b711b34b7c833d6e4393672d06bcd7c1b07c37406877"
Nov 01 06:20:52.995147 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.995122    5619 scope.go:122] "RemoveContainer" containerID="4363063ae8c674dde239f41bf33a9abaa58b11ea76e09d816dc52248f52310d7"
Nov 01 06:20:52.995489 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:20:52.995469    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4363063ae8c674dde239f41bf33a9abaa58b11ea76e09d816dc52248f52310d7\": container with ID starting with 4363063ae8c674dde239f41bf33a9abaa58b11ea76e09d816dc52248f52310d7 not found: ID does not exist" containerID="4363063ae8c674dde239f41bf33a9abaa58b11ea76e09d816dc52248f52310d7"
Nov 01 06:20:52.995551 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.995498    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4363063ae8c674dde239f41bf33a9abaa58b11ea76e09d816dc52248f52310d7"} err="failed to get container status \"4363063ae8c674dde239f41bf33a9abaa58b11ea76e09d816dc52248f52310d7\": rpc error: code = NotFound desc = could not find container \"4363063ae8c674dde239f41bf33a9abaa58b11ea76e09d816dc52248f52310d7\": container with ID starting with 4363063ae8c674dde239f41bf33a9abaa58b11ea76e09d816dc52248f52310d7 not found: ID does not exist"
Nov 01 06:20:52.995551 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.995517    5619 scope.go:122] "RemoveContainer" containerID="66bc50bf8e637149ede5b711b34b7c833d6e4393672d06bcd7c1b07c37406877"
Nov 01 06:20:52.995792 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:20:52.995763    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66bc50bf8e637149ede5b711b34b7c833d6e4393672d06bcd7c1b07c37406877\": container with ID starting with 66bc50bf8e637149ede5b711b34b7c833d6e4393672d06bcd7c1b07c37406877 not found: ID does not exist" containerID="66bc50bf8e637149ede5b711b34b7c833d6e4393672d06bcd7c1b07c37406877"
Nov 01 06:20:52.995831 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:52.995804    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66bc50bf8e637149ede5b711b34b7c833d6e4393672d06bcd7c1b07c37406877"} err="failed to get container status \"66bc50bf8e637149ede5b711b34b7c833d6e4393672d06bcd7c1b07c37406877\": rpc error: code = NotFound desc = could not find container \"66bc50bf8e637149ede5b711b34b7c833d6e4393672d06bcd7c1b07c37406877\": container with ID starting with 66bc50bf8e637149ede5b711b34b7c833d6e4393672d06bcd7c1b07c37406877 not found: ID does not exist"
Nov 01 06:20:53.126121 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:53.126082    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"]
Nov 01 06:20:53.130905 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:53.130880    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2a4cda2f-openshift-default-6468b89ff5-fvgmt"]
Nov 01 06:20:54.963592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:54.963564    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93542be2-aa8a-4d1f-ade6-33dd834c5758" path="/var/lib/kubelet/pods/93542be2-aa8a-4d1f-ade6-33dd834c5758/volumes"
Nov 01 06:20:55.547194 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:55.547138    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:55.724483 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:55.724393    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:55.910395 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:20:55.910310    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:20:55.981389 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:55.981270    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6"]
Nov 01 06:20:55.981883 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:55.981865    5619 cpu_manager.go:414] "containerMap: removing container" podUID="93542be2-aa8a-4d1f-ade6-33dd834c5758" containerName="istio-proxy"
Nov 01 06:20:55.981883 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:55.981883    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93542be2-aa8a-4d1f-ade6-33dd834c5758" containerName="istio-proxy"
Nov 01 06:20:55.982021 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:55.982014    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="93542be2-aa8a-4d1f-ade6-33dd834c5758" containerName="istio-proxy"
Nov 01 06:20:55.985496 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:55.985475    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6"
Nov 01 06:20:56.047337 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.047291    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb9z7\" (UniqueName: \"kubernetes.io/projected/127d9002-4a73-426b-bf44-453b69e6492a-kube-api-access-wb9z7\") pod \"payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6\" (UID: \"127d9002-4a73-426b-bf44-453b69e6492a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6"
Nov 01 06:20:56.047537 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.047514    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/127d9002-4a73-426b-bf44-453b69e6492a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6\" (UID: \"127d9002-4a73-426b-bf44-453b69e6492a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6"
Nov 01 06:20:56.091550 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.091515    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e5d7100d\""
Nov 01 06:20:56.094186 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.094155    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6"]
Nov 01 06:20:56.115772 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.115747    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm"]
Nov 01 06:20:56.116352 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.116313    5619 cpu_manager.go:414] "containerMap: removing container" podUID="93542be2-aa8a-4d1f-ade6-33dd834c5758" containerName="istio-proxy"
Nov 01 06:20:56.116352 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.116345    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93542be2-aa8a-4d1f-ade6-33dd834c5758" containerName="istio-proxy"
Nov 01 06:20:56.116507 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.116474    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="93542be2-aa8a-4d1f-ade6-33dd834c5758" containerName="istio-proxy"
Nov 01 06:20:56.119771 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.119755    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm"
Nov 01 06:20:56.149451 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.148351    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb9z7\" (UniqueName: \"kubernetes.io/projected/127d9002-4a73-426b-bf44-453b69e6492a-kube-api-access-wb9z7\") pod \"payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6\" (UID: \"127d9002-4a73-426b-bf44-453b69e6492a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6"
Nov 01 06:20:56.149451 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.148606    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/127d9002-4a73-426b-bf44-453b69e6492a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6\" (UID: \"127d9002-4a73-426b-bf44-453b69e6492a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6"
Nov 01 06:20:56.149743 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.149578    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/127d9002-4a73-426b-bf44-453b69e6492a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6\" (UID: \"127d9002-4a73-426b-bf44-453b69e6492a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6"
Nov 01 06:20:56.232422 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.210989    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm"]
Nov 01 06:20:56.232422 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.211198    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e5d7100d-dockercfg-xd42r\""
Nov 01 06:20:56.232422 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.228607    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb9z7\" (UniqueName: \"kubernetes.io/projected/127d9002-4a73-426b-bf44-453b69e6492a-kube-api-access-wb9z7\") pod \"payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6\" (UID: \"127d9002-4a73-426b-bf44-453b69e6492a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6"
Nov 01 06:20:56.249774 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.249725    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llw8l\" (UniqueName: \"kubernetes.io/projected/dd69d0eb-1c30-4eb9-adec-145fd01ebeb4-kube-api-access-llw8l\") pod \"payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm\" (UID: \"dd69d0eb-1c30-4eb9-adec-145fd01ebeb4\") " pod="openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm"
Nov 01 06:20:56.249953 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.249892    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd69d0eb-1c30-4eb9-adec-145fd01ebeb4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm\" (UID: \"dd69d0eb-1c30-4eb9-adec-145fd01ebeb4\") " pod="openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm"
Nov 01 06:20:56.299482 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.299459    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6"
Nov 01 06:20:56.351472 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.351439    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llw8l\" (UniqueName: \"kubernetes.io/projected/dd69d0eb-1c30-4eb9-adec-145fd01ebeb4-kube-api-access-llw8l\") pod \"payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm\" (UID: \"dd69d0eb-1c30-4eb9-adec-145fd01ebeb4\") " pod="openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm"
Nov 01 06:20:56.351650 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.351533    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd69d0eb-1c30-4eb9-adec-145fd01ebeb4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm\" (UID: \"dd69d0eb-1c30-4eb9-adec-145fd01ebeb4\") " pod="openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm"
Nov 01 06:20:56.352496 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.352435    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd69d0eb-1c30-4eb9-adec-145fd01ebeb4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm\" (UID: \"dd69d0eb-1c30-4eb9-adec-145fd01ebeb4\") " pod="openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm"
Nov 01 06:20:56.414863 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.414822    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llw8l\" (UniqueName: \"kubernetes.io/projected/dd69d0eb-1c30-4eb9-adec-145fd01ebeb4-kube-api-access-llw8l\") pod \"payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm\" (UID: \"dd69d0eb-1c30-4eb9-adec-145fd01ebeb4\") " pod="openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm"
Nov 01 06:20:56.535649 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.535568    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm"
Nov 01 06:20:56.678686 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.678663    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6"]
Nov 01 06:20:56.680866 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:20:56.680836    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod127d9002_4a73_426b_bf44_453b69e6492a.slice/crio-a6113870205661a51b5cbe09e819f91ee10ce1034c1bad5a9a97963dd9eb8ab1 WatchSource:0}: Error finding container a6113870205661a51b5cbe09e819f91ee10ce1034c1bad5a9a97963dd9eb8ab1: Status 404 returned error can't find the container with id a6113870205661a51b5cbe09e819f91ee10ce1034c1bad5a9a97963dd9eb8ab1
Nov 01 06:20:56.919825 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.919663    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm"]
Nov 01 06:20:56.923084 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:20:56.923045    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd69d0eb_1c30_4eb9_adec_145fd01ebeb4.slice/crio-188679ca65bf0afb782197c0cc5fad1d363ff6ad33b0e129e547818afc27ac0d WatchSource:0}: Error finding container 188679ca65bf0afb782197c0cc5fad1d363ff6ad33b0e129e547818afc27ac0d: Status 404 returned error can't find the container with id 188679ca65bf0afb782197c0cc5fad1d363ff6ad33b0e129e547818afc27ac0d
Nov 01 06:20:56.985742 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.985715    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm" event={"ID":"dd69d0eb-1c30-4eb9-adec-145fd01ebeb4","Type":"ContainerStarted","Data":"188679ca65bf0afb782197c0cc5fad1d363ff6ad33b0e129e547818afc27ac0d"}
Nov 01 06:20:56.987652 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.987624    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6" event={"ID":"127d9002-4a73-426b-bf44-453b69e6492a","Type":"ContainerStarted","Data":"6fde6574d2b5057284801cbf362112a62ae014b864094fcf64cf098f748cc077"}
Nov 01 06:20:56.987768 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.987691    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6" event={"ID":"127d9002-4a73-426b-bf44-453b69e6492a","Type":"ContainerStarted","Data":"a6113870205661a51b5cbe09e819f91ee10ce1034c1bad5a9a97963dd9eb8ab1"}
Nov 01 06:20:56.987768 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:56.987740    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6"
Nov 01 06:20:57.111982 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:57.111898    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6" podStartSLOduration=2.111876856 podStartE2EDuration="2.111876856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:20:57.107813148 +0000 UTC m=+2094.318636658" watchObservedRunningTime="2026-11-01 06:20:57.111876856 +0000 UTC m=+2094.322700365"
Nov 01 06:20:57.296928 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:57.296842    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:20:57.995474 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:57.995432    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm" event={"ID":"dd69d0eb-1c30-4eb9-adec-145fd01ebeb4","Type":"ContainerStarted","Data":"6c85cc33bddd0b48de08dcd162cbbaf09b6e13d150d940668c2341cd1e729aba"}
Nov 01 06:20:57.995957 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:57.995722    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm"
Nov 01 06:20:58.115005 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:20:58.114951    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm" podStartSLOduration=2.11493794 podStartE2EDuration="2.11493794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:20:58.112777571 +0000 UTC m=+2095.323601081" watchObservedRunningTime="2026-11-01 06:20:58.11493794 +0000 UTC m=+2095.325761462"
Nov 01 06:21:01.979954 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:01.979916    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:06.682614 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:06.682574    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:07.998149 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:07.998120    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6"
Nov 01 06:21:09.005831 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:09.005802    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm"
Nov 01 06:21:11.472217 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:11.472176    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:16.313844 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:16.313805    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:21.065670 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:21.065633    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:23.481766 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:23.481726    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:25.417376 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:25.417311    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:25.576241 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:25.576180    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:25.771141 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:25.771053    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:25.879805 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:21:25.879773    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:21:28.279417 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:28.279384    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:30.571079 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:30.571032    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:31.590451 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:31.590413    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:35.270363 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:35.270305    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:38.316569 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:38.316531    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:39.704742 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:39.704699    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"]
Nov 01 06:21:39.709126 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:39.709103    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"
Nov 01 06:21:39.815821 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:39.815785    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-e5d7100d-serving-cert\""
Nov 01 06:21:39.818492 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:39.818462    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"]
Nov 01 06:21:39.861868 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:39.861830    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0e5170-1714-4595-b348-1ee089572a62-openshift-service-ca\") pod \"maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk\" (UID: \"7c0e5170-1714-4595-b348-1ee089572a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"
Nov 01 06:21:39.862039 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:39.862014    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c0e5170-1714-4595-b348-1ee089572a62-maas-api-tls\") pod \"maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk\" (UID: \"7c0e5170-1714-4595-b348-1ee089572a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"
Nov 01 06:21:39.862089 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:39.862061    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c0e5170-1714-4595-b348-1ee089572a62-metrics-certs\") pod \"maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk\" (UID: \"7c0e5170-1714-4595-b348-1ee089572a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"
Nov 01 06:21:39.862240 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:39.862201    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp9lt\" (UniqueName: \"kubernetes.io/projected/7c0e5170-1714-4595-b348-1ee089572a62-kube-api-access-mp9lt\") pod \"maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk\" (UID: \"7c0e5170-1714-4595-b348-1ee089572a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"
Nov 01 06:21:39.896614 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:39.896576    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:39.964277 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:39.964172    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c0e5170-1714-4595-b348-1ee089572a62-metrics-certs\") pod \"maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk\" (UID: \"7c0e5170-1714-4595-b348-1ee089572a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"
Nov 01 06:21:39.964503 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:39.964290    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp9lt\" (UniqueName: \"kubernetes.io/projected/7c0e5170-1714-4595-b348-1ee089572a62-kube-api-access-mp9lt\") pod \"maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk\" (UID: \"7c0e5170-1714-4595-b348-1ee089572a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"
Nov 01 06:21:39.964503 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:39.964389    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0e5170-1714-4595-b348-1ee089572a62-openshift-service-ca\") pod \"maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk\" (UID: \"7c0e5170-1714-4595-b348-1ee089572a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"
Nov 01 06:21:39.964650 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:39.964626    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c0e5170-1714-4595-b348-1ee089572a62-maas-api-tls\") pod \"maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk\" (UID: \"7c0e5170-1714-4595-b348-1ee089572a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"
Nov 01 06:21:39.964820 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:21:39.964800    5619 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-serving-cert: secret "maas-api-e2e-migrate-e5d7100d-serving-cert" not found
Nov 01 06:21:39.964894 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:21:39.964882    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c0e5170-1714-4595-b348-1ee089572a62-maas-api-tls podName:7c0e5170-1714-4595-b348-1ee089572a62 nodeName:}" failed. No retries permitted until 2026-11-01 06:21:40.464861886 +0000 UTC m=+2137.675685373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c0e5170-1714-4595-b348-1ee089572a62-maas-api-tls") pod "maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk" (UID: "7c0e5170-1714-4595-b348-1ee089572a62") : secret "maas-api-e2e-migrate-e5d7100d-serving-cert" not found
Nov 01 06:21:39.965711 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:39.965687    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0e5170-1714-4595-b348-1ee089572a62-openshift-service-ca\") pod \"maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk\" (UID: \"7c0e5170-1714-4595-b348-1ee089572a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"
Nov 01 06:21:39.967544 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:39.967518    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c0e5170-1714-4595-b348-1ee089572a62-metrics-certs\") pod \"maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk\" (UID: \"7c0e5170-1714-4595-b348-1ee089572a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"
Nov 01 06:21:40.028997 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:40.028962    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp9lt\" (UniqueName: \"kubernetes.io/projected/7c0e5170-1714-4595-b348-1ee089572a62-kube-api-access-mp9lt\") pod \"maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk\" (UID: \"7c0e5170-1714-4595-b348-1ee089572a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"
Nov 01 06:21:40.085836 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:40.085802    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:40.303647 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:40.303560    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:40.470315 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:40.470279    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c0e5170-1714-4595-b348-1ee089572a62-maas-api-tls\") pod \"maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk\" (UID: \"7c0e5170-1714-4595-b348-1ee089572a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"
Nov 01 06:21:40.472705 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:40.472679    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c0e5170-1714-4595-b348-1ee089572a62-maas-api-tls\") pod \"maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk\" (UID: \"7c0e5170-1714-4595-b348-1ee089572a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"
Nov 01 06:21:40.623655 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:40.623625    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"
Nov 01 06:21:41.002644 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:41.002614    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"]
Nov 01 06:21:41.003451 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:21:41.003423    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0e5170_1714_4595_b348_1ee089572a62.slice/crio-997fc717de7f8eec82e174ddb700683b0808184be3530e3aa343486408f10c51 WatchSource:0}: Error finding container 997fc717de7f8eec82e174ddb700683b0808184be3530e3aa343486408f10c51: Status 404 returned error can't find the container with id 997fc717de7f8eec82e174ddb700683b0808184be3530e3aa343486408f10c51
Nov 01 06:21:41.270907 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:41.270822    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk" event={"ID":"7c0e5170-1714-4595-b348-1ee089572a62","Type":"ContainerStarted","Data":"997fc717de7f8eec82e174ddb700683b0808184be3530e3aa343486408f10c51"}
Nov 01 06:21:42.278600 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:42.278496    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk" event={"ID":"7c0e5170-1714-4595-b348-1ee089572a62","Type":"ContainerStarted","Data":"ff6fc43e9cdc59c972e1bf299caef45fa6d3c76e6aa632529fa2fa137f9899dd"}
Nov 01 06:21:42.278600 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:42.278584    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"
Nov 01 06:21:44.482570 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:44.482522    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:48.294334 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:48.294283    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"
Nov 01 06:21:48.416543 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:48.416490    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk" podStartSLOduration=8.526150695 podStartE2EDuration="9.416478245s" totalImagesPullingTime="890.32755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:21:42.397494275 +0000 UTC m=+2139.608317785" watchObservedRunningTime="2026-11-01 06:21:48.416478245 +0000 UTC m=+2145.627301748"
Nov 01 06:21:49.359106 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:49.359070    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:54.065860 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:54.065819    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:21:55.880794 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:21:55.880753    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:21:58.847557 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:21:58.847500    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:22:03.791790 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:03.791741    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:22:06.986542 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:06.986506    5619 patch_prober.go:29] interesting pod/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:22:06.986957 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:06.986585    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2" podUID="cb80df79-1ef0-4be1-8607-0394242a387a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:22:08.570687 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:08.570631    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:22:09.497567 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:09.497531    5619 generic.go:356] "Generic (PLEG): container finished" podID="cb80df79-1ef0-4be1-8607-0394242a387a" containerID="b198e622e33cb9aa30a4246239a00504ab2f3c26cc77339c73a3b94e60995126" exitCode=0
Nov 01 06:22:09.497814 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:09.497601    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2" event={"ID":"cb80df79-1ef0-4be1-8607-0394242a387a","Type":"ContainerDied","Data":"b198e622e33cb9aa30a4246239a00504ab2f3c26cc77339c73a3b94e60995126"}
Nov 01 06:22:09.498072 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:09.498054    5619 scope.go:122] "RemoveContainer" containerID="b198e622e33cb9aa30a4246239a00504ab2f3c26cc77339c73a3b94e60995126"
Nov 01 06:22:09.499664 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:09.499633    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:22:09.499751 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:09.499708    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:22:09.499796 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:09.499745    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:22:10.506263 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:10.506232    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2" event={"ID":"cb80df79-1ef0-4be1-8607-0394242a387a","Type":"ContainerStarted","Data":"66035ff2b9c73d6187591bcd22212cb257fac88e6bf595cbfb8c544ce65c1c43"}
Nov 01 06:22:11.263348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:11.257715    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:22:11.263348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:11.257834    5619 patch_prober.go:29] interesting pod/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 01 06:22:11.263348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:11.257890    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2" podUID="cb80df79-1ef0-4be1-8607-0394242a387a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 01 06:22:12.259721 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:12.259682    5619 patch_prober.go:29] interesting pod/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 01 06:22:12.260304 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:12.260270    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2" podUID="cb80df79-1ef0-4be1-8607-0394242a387a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 01 06:22:13.273346 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:13.273281    5619 patch_prober.go:29] interesting pod/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 01 06:22:13.273854 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:13.273366    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2" podUID="cb80df79-1ef0-4be1-8607-0394242a387a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 01 06:22:13.411341 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:13.411287    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:22:14.257736 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:14.257694    5619 patch_prober.go:29] interesting pod/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 01 06:22:14.258026 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:14.257998    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2" podUID="cb80df79-1ef0-4be1-8607-0394242a387a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 01 06:22:15.261348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:15.258699    5619 patch_prober.go:29] interesting pod/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 01 06:22:15.261348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:15.258762    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2" podUID="cb80df79-1ef0-4be1-8607-0394242a387a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 01 06:22:16.261533 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:16.261501    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:22:16.261952 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:16.261800    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:22:16.262500 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:16.262474    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:22:18.310246 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:18.310212    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:22:23.066763 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:23.066713    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:22:23.827312 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:23.827277    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:22:25.880787 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:22:25.880751    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:22:27.795071 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:27.795032    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:22:28.629388 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:28.629357    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:22:32.470829 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:32.470790    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:22:36.949015 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:36.948896    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:22:42.062707 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:42.062665    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:22:46.788150 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:46.788117    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:22:51.625626 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:51.625579    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:22:55.881213 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:22:55.881170    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:22:56.577071 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:56.577012    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:22:57.278331 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:57.278286    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"]
Nov 01 06:22:57.278744 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:57.278498    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg" podUID="929d43c9-ec49-4988-a7af-ad7f49fea0ff" containerName="istio-proxy" containerID="cri-o://c0bb602ee87978d53259695eb806cbffd2697611b51cdf51d48f5a252e2ad679" gracePeriod=30
Nov 01 06:22:57.831318 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:57.831284    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:22:57.831490 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:22:57.831371    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg" podUID="929d43c9-ec49-4988-a7af-ad7f49fea0ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:23:01.190400 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:01.190363    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm"]
Nov 01 06:23:01.298298 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:01.298267    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6"]
Nov 01 06:23:01.364758 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:01.364713    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:01.567061 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:01.566959    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:01.715703 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:01.715668    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:02.283592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.283516    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:02.550864 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.550798    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:23:02.601251 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.601220    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929d43c9-ec49-4988-a7af-ad7f49fea0ff-kube-api-access-sg78x\" (UniqueName: \"kubernetes.io/projected/929d43c9-ec49-4988-a7af-ad7f49fea0ff-kube-api-access-sg78x\") pod \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") "
Nov 01 06:23:02.601441 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.601331    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-workload-socket\") pod \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") "
Nov 01 06:23:02.601441 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.601384    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-podinfo\") pod \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") "
Nov 01 06:23:02.601578 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.601445    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-data\") pod \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") "
Nov 01 06:23:02.601578 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.601523    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-workload-certs\") pod \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") "
Nov 01 06:23:02.601578 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.601556    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-envoy\") pod \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") "
Nov 01 06:23:02.601747 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.601587    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-token\" (UniqueName: \"kubernetes.io/projected/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-token\") pod \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") "
Nov 01 06:23:02.601747 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.601596    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-workload-socket" pod "929d43c9-ec49-4988-a7af-ad7f49fea0ff" (UID: "929d43c9-ec49-4988-a7af-ad7f49fea0ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:23:02.601747 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.601633    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-credential-socket\") pod \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") "
Nov 01 06:23:02.601747 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.601674    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istiod-ca-cert\") pod \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\" (UID: \"929d43c9-ec49-4988-a7af-ad7f49fea0ff\") "
Nov 01 06:23:02.601970 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.601745    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-data" pod "929d43c9-ec49-4988-a7af-ad7f49fea0ff" (UID: "929d43c9-ec49-4988-a7af-ad7f49fea0ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:23:02.601970 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.601762    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-workload-certs" pod "929d43c9-ec49-4988-a7af-ad7f49fea0ff" (UID: "929d43c9-ec49-4988-a7af-ad7f49fea0ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:23:02.601970 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.601962    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-credential-socket" pod "929d43c9-ec49-4988-a7af-ad7f49fea0ff" (UID: "929d43c9-ec49-4988-a7af-ad7f49fea0ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:23:02.602195 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.602157    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istiod-ca-cert" pod "929d43c9-ec49-4988-a7af-ad7f49fea0ff" (UID: "929d43c9-ec49-4988-a7af-ad7f49fea0ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:23:02.602283 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.602259    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:02.602369 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.602283    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:02.602369 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.602292    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:02.602369 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.602301    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:02.602369 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.602309    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:02.603697 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.603674    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-podinfo" pod "929d43c9-ec49-4988-a7af-ad7f49fea0ff" (UID: "929d43c9-ec49-4988-a7af-ad7f49fea0ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:23:02.603697 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.603689    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-envoy" pod "929d43c9-ec49-4988-a7af-ad7f49fea0ff" (UID: "929d43c9-ec49-4988-a7af-ad7f49fea0ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:23:02.604105 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.604080    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929d43c9-ec49-4988-a7af-ad7f49fea0ff-kube-api-access-sg78x" pod "929d43c9-ec49-4988-a7af-ad7f49fea0ff" (UID: "929d43c9-ec49-4988-a7af-ad7f49fea0ff"). InnerVolumeSpecName "kube-api-access-sg78x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:23:02.604105 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.604075    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-token" pod "929d43c9-ec49-4988-a7af-ad7f49fea0ff" (UID: "929d43c9-ec49-4988-a7af-ad7f49fea0ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:23:02.704013 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.703974    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg78x\" (UniqueName: \"kubernetes.io/projected/929d43c9-ec49-4988-a7af-ad7f49fea0ff-kube-api-access-sg78x\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:02.704013 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.704016    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:02.704206 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.704032    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:02.704206 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.704046    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929d43c9-ec49-4988-a7af-ad7f49fea0ff-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:02.824521 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.824402    5619 generic.go:356] "Generic (PLEG): container finished" podID="929d43c9-ec49-4988-a7af-ad7f49fea0ff" containerID="c0bb602ee87978d53259695eb806cbffd2697611b51cdf51d48f5a252e2ad679" exitCode=0
Nov 01 06:23:02.824521 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.824466    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"
Nov 01 06:23:02.824521 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.824493    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg" event={"ID":"929d43c9-ec49-4988-a7af-ad7f49fea0ff","Type":"ContainerDied","Data":"c0bb602ee87978d53259695eb806cbffd2697611b51cdf51d48f5a252e2ad679"}
Nov 01 06:23:02.824789 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.824559    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg" event={"ID":"929d43c9-ec49-4988-a7af-ad7f49fea0ff","Type":"ContainerDied","Data":"8d72ce0aeef59fe3b501fb259c0d57ef77b55d85e19bf982c5834fcfcd8f155d"}
Nov 01 06:23:02.824789 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.824591    5619 scope.go:122] "RemoveContainer" containerID="c0bb602ee87978d53259695eb806cbffd2697611b51cdf51d48f5a252e2ad679"
Nov 01 06:23:02.837521 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.837496    5619 scope.go:122] "RemoveContainer" containerID="c0bb602ee87978d53259695eb806cbffd2697611b51cdf51d48f5a252e2ad679"
Nov 01 06:23:02.837801 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:23:02.837774    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0bb602ee87978d53259695eb806cbffd2697611b51cdf51d48f5a252e2ad679\": container with ID starting with c0bb602ee87978d53259695eb806cbffd2697611b51cdf51d48f5a252e2ad679 not found: ID does not exist" containerID="c0bb602ee87978d53259695eb806cbffd2697611b51cdf51d48f5a252e2ad679"
Nov 01 06:23:02.837877 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.837802    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0bb602ee87978d53259695eb806cbffd2697611b51cdf51d48f5a252e2ad679"} err="failed to get container status \"c0bb602ee87978d53259695eb806cbffd2697611b51cdf51d48f5a252e2ad679\": rpc error: code = NotFound desc = could not find container \"c0bb602ee87978d53259695eb806cbffd2697611b51cdf51d48f5a252e2ad679\": container with ID starting with c0bb602ee87978d53259695eb806cbffd2697611b51cdf51d48f5a252e2ad679 not found: ID does not exist"
Nov 01 06:23:02.885345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.885000    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:02.993120 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.993092    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"]
Nov 01 06:23:02.997124 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:02.997099    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-vxtcg"]
Nov 01 06:23:03.084927 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:03.084745    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:04.964379 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:04.964345    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="929d43c9-ec49-4988-a7af-ad7f49fea0ff" path="/var/lib/kubelet/pods/929d43c9-ec49-4988-a7af-ad7f49fea0ff/volumes"
Nov 01 06:23:05.510795 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:05.510758    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:05.806594 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:05.806498    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:06.191384 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.191350    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm" podUID="dd69d0eb-1c30-4eb9-adec-145fd01ebeb4" containerName="payload-processing" containerID="cri-o://6c85cc33bddd0b48de08dcd162cbbaf09b6e13d150d940668c2341cd1e729aba" gracePeriod=25
Nov 01 06:23:06.279770 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.279730    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:06.300630 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.300557    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6" podUID="127d9002-4a73-426b-bf44-453b69e6492a" containerName="payload-pre-processing" containerID="cri-o://6fde6574d2b5057284801cbf362112a62ae014b864094fcf64cf098f748cc077" gracePeriod=25
Nov 01 06:23:06.482136 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.482096    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm"
Nov 01 06:23:06.547950 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.547921    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd69d0eb-1c30-4eb9-adec-145fd01ebeb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd69d0eb-1c30-4eb9-adec-145fd01ebeb4-plugins-config-volume\") pod \"dd69d0eb-1c30-4eb9-adec-145fd01ebeb4\" (UID: \"dd69d0eb-1c30-4eb9-adec-145fd01ebeb4\") "
Nov 01 06:23:06.548120 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.547973    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd69d0eb-1c30-4eb9-adec-145fd01ebeb4-kube-api-access-llw8l\" (UniqueName: \"kubernetes.io/projected/dd69d0eb-1c30-4eb9-adec-145fd01ebeb4-kube-api-access-llw8l\") pod \"dd69d0eb-1c30-4eb9-adec-145fd01ebeb4\" (UID: \"dd69d0eb-1c30-4eb9-adec-145fd01ebeb4\") "
Nov 01 06:23:06.548405 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.548376    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd69d0eb-1c30-4eb9-adec-145fd01ebeb4-plugins-config-volume" pod "dd69d0eb-1c30-4eb9-adec-145fd01ebeb4" (UID: "dd69d0eb-1c30-4eb9-adec-145fd01ebeb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:23:06.548593 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.548575    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd69d0eb-1c30-4eb9-adec-145fd01ebeb4-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:06.551497 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.551473    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd69d0eb-1c30-4eb9-adec-145fd01ebeb4-kube-api-access-llw8l" pod "dd69d0eb-1c30-4eb9-adec-145fd01ebeb4" (UID: "dd69d0eb-1c30-4eb9-adec-145fd01ebeb4"). InnerVolumeSpecName "kube-api-access-llw8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:23:06.572838 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.572808    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6"
Nov 01 06:23:06.649449 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.649413    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/127d9002-4a73-426b-bf44-453b69e6492a-kube-api-access-wb9z7\" (UniqueName: \"kubernetes.io/projected/127d9002-4a73-426b-bf44-453b69e6492a-kube-api-access-wb9z7\") pod \"127d9002-4a73-426b-bf44-453b69e6492a\" (UID: \"127d9002-4a73-426b-bf44-453b69e6492a\") "
Nov 01 06:23:06.649627 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.649563    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/127d9002-4a73-426b-bf44-453b69e6492a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/127d9002-4a73-426b-bf44-453b69e6492a-plugins-config-volume\") pod \"127d9002-4a73-426b-bf44-453b69e6492a\" (UID: \"127d9002-4a73-426b-bf44-453b69e6492a\") "
Nov 01 06:23:06.649985 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.649963    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llw8l\" (UniqueName: \"kubernetes.io/projected/dd69d0eb-1c30-4eb9-adec-145fd01ebeb4-kube-api-access-llw8l\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:06.650115 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.649987    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/127d9002-4a73-426b-bf44-453b69e6492a-plugins-config-volume" pod "127d9002-4a73-426b-bf44-453b69e6492a" (UID: "127d9002-4a73-426b-bf44-453b69e6492a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:23:06.651930 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.651900    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:06.652052 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.652030    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/127d9002-4a73-426b-bf44-453b69e6492a-kube-api-access-wb9z7" pod "127d9002-4a73-426b-bf44-453b69e6492a" (UID: "127d9002-4a73-426b-bf44-453b69e6492a"). InnerVolumeSpecName "kube-api-access-wb9z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:23:06.751998 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.751906    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb9z7\" (UniqueName: \"kubernetes.io/projected/127d9002-4a73-426b-bf44-453b69e6492a-kube-api-access-wb9z7\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:06.751998 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.751946    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/127d9002-4a73-426b-bf44-453b69e6492a-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:06.815237 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.815207    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:06.849662 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.849623    5619 generic.go:356] "Generic (PLEG): container finished" podID="dd69d0eb-1c30-4eb9-adec-145fd01ebeb4" containerID="6c85cc33bddd0b48de08dcd162cbbaf09b6e13d150d940668c2341cd1e729aba" exitCode=0
Nov 01 06:23:06.849662 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.849654    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm" event={"ID":"dd69d0eb-1c30-4eb9-adec-145fd01ebeb4","Type":"ContainerDied","Data":"6c85cc33bddd0b48de08dcd162cbbaf09b6e13d150d940668c2341cd1e729aba"}
Nov 01 06:23:06.849890 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.849735    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm" event={"ID":"dd69d0eb-1c30-4eb9-adec-145fd01ebeb4","Type":"ContainerDied","Data":"188679ca65bf0afb782197c0cc5fad1d363ff6ad33b0e129e547818afc27ac0d"}
Nov 01 06:23:06.849890 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.849667    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm"
Nov 01 06:23:06.849890 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.849761    5619 scope.go:122] "RemoveContainer" containerID="6c85cc33bddd0b48de08dcd162cbbaf09b6e13d150d940668c2341cd1e729aba"
Nov 01 06:23:06.851466 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.851440    5619 generic.go:356] "Generic (PLEG): container finished" podID="127d9002-4a73-426b-bf44-453b69e6492a" containerID="6fde6574d2b5057284801cbf362112a62ae014b864094fcf64cf098f748cc077" exitCode=0
Nov 01 06:23:06.851604 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.851495    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6"
Nov 01 06:23:06.851604 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.851516    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6" event={"ID":"127d9002-4a73-426b-bf44-453b69e6492a","Type":"ContainerDied","Data":"6fde6574d2b5057284801cbf362112a62ae014b864094fcf64cf098f748cc077"}
Nov 01 06:23:06.851604 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.851573    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6" event={"ID":"127d9002-4a73-426b-bf44-453b69e6492a","Type":"ContainerDied","Data":"a6113870205661a51b5cbe09e819f91ee10ce1034c1bad5a9a97963dd9eb8ab1"}
Nov 01 06:23:06.862539 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.862519    5619 scope.go:122] "RemoveContainer" containerID="6c85cc33bddd0b48de08dcd162cbbaf09b6e13d150d940668c2341cd1e729aba"
Nov 01 06:23:06.862852 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:23:06.862825    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c85cc33bddd0b48de08dcd162cbbaf09b6e13d150d940668c2341cd1e729aba\": container with ID starting with 6c85cc33bddd0b48de08dcd162cbbaf09b6e13d150d940668c2341cd1e729aba not found: ID does not exist" containerID="6c85cc33bddd0b48de08dcd162cbbaf09b6e13d150d940668c2341cd1e729aba"
Nov 01 06:23:06.862945 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.862864    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c85cc33bddd0b48de08dcd162cbbaf09b6e13d150d940668c2341cd1e729aba"} err="failed to get container status \"6c85cc33bddd0b48de08dcd162cbbaf09b6e13d150d940668c2341cd1e729aba\": rpc error: code = NotFound desc = could not find container \"6c85cc33bddd0b48de08dcd162cbbaf09b6e13d150d940668c2341cd1e729aba\": container with ID starting with 6c85cc33bddd0b48de08dcd162cbbaf09b6e13d150d940668c2341cd1e729aba not found: ID does not exist"
Nov 01 06:23:06.862945 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.862892    5619 scope.go:122] "RemoveContainer" containerID="6fde6574d2b5057284801cbf362112a62ae014b864094fcf64cf098f748cc077"
Nov 01 06:23:06.871714 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.871696    5619 scope.go:122] "RemoveContainer" containerID="6fde6574d2b5057284801cbf362112a62ae014b864094fcf64cf098f748cc077"
Nov 01 06:23:06.871973 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:23:06.871957    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fde6574d2b5057284801cbf362112a62ae014b864094fcf64cf098f748cc077\": container with ID starting with 6fde6574d2b5057284801cbf362112a62ae014b864094fcf64cf098f748cc077 not found: ID does not exist" containerID="6fde6574d2b5057284801cbf362112a62ae014b864094fcf64cf098f748cc077"
Nov 01 06:23:06.872043 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.871979    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fde6574d2b5057284801cbf362112a62ae014b864094fcf64cf098f748cc077"} err="failed to get container status \"6fde6574d2b5057284801cbf362112a62ae014b864094fcf64cf098f748cc077\": rpc error: code = NotFound desc = could not find container \"6fde6574d2b5057284801cbf362112a62ae014b864094fcf64cf098f748cc077\": container with ID starting with 6fde6574d2b5057284801cbf362112a62ae014b864094fcf64cf098f748cc077 not found: ID does not exist"
Nov 01 06:23:06.961827 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:06.961797    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:07.016957 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:07.016888    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6"]
Nov 01 06:23:07.021268 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:07.021245    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e5d7100d-5975bffd4c-rfzj6"]
Nov 01 06:23:07.132438 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:07.132408    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm"]
Nov 01 06:23:07.138166 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:07.138143    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e5d7100d-7ff88bd549-l5ssm"]
Nov 01 06:23:08.963385 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:08.963356    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="127d9002-4a73-426b-bf44-453b69e6492a" path="/var/lib/kubelet/pods/127d9002-4a73-426b-bf44-453b69e6492a/volumes"
Nov 01 06:23:08.963805 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:08.963791    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd69d0eb-1c30-4eb9-adec-145fd01ebeb4" path="/var/lib/kubelet/pods/dd69d0eb-1c30-4eb9-adec-145fd01ebeb4/volumes"
Nov 01 06:23:09.853096 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:09.853059    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:10.048633 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.048596    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:10.262026 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.261937    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266"]
Nov 01 06:23:10.262632 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.262610    5619 cpu_manager.go:414] "containerMap: removing container" podUID="dd69d0eb-1c30-4eb9-adec-145fd01ebeb4" containerName="payload-processing"
Nov 01 06:23:10.262632 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.262632    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd69d0eb-1c30-4eb9-adec-145fd01ebeb4" containerName="payload-processing"
Nov 01 06:23:10.262812 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.262655    5619 cpu_manager.go:414] "containerMap: removing container" podUID="929d43c9-ec49-4988-a7af-ad7f49fea0ff" containerName="istio-proxy"
Nov 01 06:23:10.262812 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.262662    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="929d43c9-ec49-4988-a7af-ad7f49fea0ff" containerName="istio-proxy"
Nov 01 06:23:10.262812 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.262687    5619 cpu_manager.go:414] "containerMap: removing container" podUID="127d9002-4a73-426b-bf44-453b69e6492a" containerName="payload-pre-processing"
Nov 01 06:23:10.262812 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.262696    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="127d9002-4a73-426b-bf44-453b69e6492a" containerName="payload-pre-processing"
Nov 01 06:23:10.263067 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.262840    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="127d9002-4a73-426b-bf44-453b69e6492a" containerName="payload-pre-processing"
Nov 01 06:23:10.263067 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.262859    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd69d0eb-1c30-4eb9-adec-145fd01ebeb4" containerName="payload-processing"
Nov 01 06:23:10.263067 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.262873    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="929d43c9-ec49-4988-a7af-ad7f49fea0ff" containerName="istio-proxy"
Nov 01 06:23:10.267728 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.267705    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266"
Nov 01 06:23:10.382033 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.381996    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c7e551a7-dockercfg-5lpgr\""
Nov 01 06:23:10.385992 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.385902    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c7e551a7\""
Nov 01 06:23:10.388385 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.388357    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266"]
Nov 01 06:23:10.409282 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.409243    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2pbr\" (UniqueName: \"kubernetes.io/projected/451b691f-2ec5-48a8-b54a-11da6c61fd64-kube-api-access-v2pbr\") pod \"payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266\" (UID: \"451b691f-2ec5-48a8-b54a-11da6c61fd64\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266"
Nov 01 06:23:10.409544 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.409427    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/451b691f-2ec5-48a8-b54a-11da6c61fd64-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266\" (UID: \"451b691f-2ec5-48a8-b54a-11da6c61fd64\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266"
Nov 01 06:23:10.435558 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.435528    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz"]
Nov 01 06:23:10.439716 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.439691    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz"
Nov 01 06:23:10.474301 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.474271    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz"]
Nov 01 06:23:10.510154 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.510128    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2pbr\" (UniqueName: \"kubernetes.io/projected/451b691f-2ec5-48a8-b54a-11da6c61fd64-kube-api-access-v2pbr\") pod \"payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266\" (UID: \"451b691f-2ec5-48a8-b54a-11da6c61fd64\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266"
Nov 01 06:23:10.510302 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.510239    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/451b691f-2ec5-48a8-b54a-11da6c61fd64-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266\" (UID: \"451b691f-2ec5-48a8-b54a-11da6c61fd64\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266"
Nov 01 06:23:10.510951 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.510926    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/451b691f-2ec5-48a8-b54a-11da6c61fd64-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266\" (UID: \"451b691f-2ec5-48a8-b54a-11da6c61fd64\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266"
Nov 01 06:23:10.572439 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.572381    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2pbr\" (UniqueName: \"kubernetes.io/projected/451b691f-2ec5-48a8-b54a-11da6c61fd64-kube-api-access-v2pbr\") pod \"payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266\" (UID: \"451b691f-2ec5-48a8-b54a-11da6c61fd64\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266"
Nov 01 06:23:10.581681 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.581654    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266"
Nov 01 06:23:10.611741 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.611710    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89tf7\" (UniqueName: \"kubernetes.io/projected/4cb4a057-3d46-441d-bb00-8057e63c9d37-kube-api-access-89tf7\") pod \"payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz\" (UID: \"4cb4a057-3d46-441d-bb00-8057e63c9d37\") " pod="openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz"
Nov 01 06:23:10.611882 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.611773    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb4a057-3d46-441d-bb00-8057e63c9d37-plugins-config-volume\") pod \"payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz\" (UID: \"4cb4a057-3d46-441d-bb00-8057e63c9d37\") " pod="openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz"
Nov 01 06:23:10.713550 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.713517    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89tf7\" (UniqueName: \"kubernetes.io/projected/4cb4a057-3d46-441d-bb00-8057e63c9d37-kube-api-access-89tf7\") pod \"payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz\" (UID: \"4cb4a057-3d46-441d-bb00-8057e63c9d37\") " pod="openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz"
Nov 01 06:23:10.713719 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.713564    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb4a057-3d46-441d-bb00-8057e63c9d37-plugins-config-volume\") pod \"payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz\" (UID: \"4cb4a057-3d46-441d-bb00-8057e63c9d37\") " pod="openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz"
Nov 01 06:23:10.714285 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.714264    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb4a057-3d46-441d-bb00-8057e63c9d37-plugins-config-volume\") pod \"payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz\" (UID: \"4cb4a057-3d46-441d-bb00-8057e63c9d37\") " pod="openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz"
Nov 01 06:23:10.775857 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.775821    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89tf7\" (UniqueName: \"kubernetes.io/projected/4cb4a057-3d46-441d-bb00-8057e63c9d37-kube-api-access-89tf7\") pod \"payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz\" (UID: \"4cb4a057-3d46-441d-bb00-8057e63c9d37\") " pod="openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz"
Nov 01 06:23:10.961219 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:23:10.960662    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451b691f_2ec5_48a8_b54a_11da6c61fd64.slice/crio-a279ba0253704794ed776e807c6a68d088e95b9f59cffdec15ce41f3ea085f4d WatchSource:0}: Error finding container a279ba0253704794ed776e807c6a68d088e95b9f59cffdec15ce41f3ea085f4d: Status 404 returned error can't find the container with id a279ba0253704794ed776e807c6a68d088e95b9f59cffdec15ce41f3ea085f4d
Nov 01 06:23:10.965630 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:10.965605    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266"]
Nov 01 06:23:11.053658 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:11.053626    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz"
Nov 01 06:23:11.426297 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:11.426268    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz"]
Nov 01 06:23:11.427211 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:23:11.427185    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb4a057_3d46_441d_bb00_8057e63c9d37.slice/crio-4549b00ba98fe9a5b3c1484e05ebacd89cbd50fa3339d4d40517f8092ea15382 WatchSource:0}: Error finding container 4549b00ba98fe9a5b3c1484e05ebacd89cbd50fa3339d4d40517f8092ea15382: Status 404 returned error can't find the container with id 4549b00ba98fe9a5b3c1484e05ebacd89cbd50fa3339d4d40517f8092ea15382
Nov 01 06:23:11.608108 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:11.608072    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:11.884584 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:11.884545    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266" event={"ID":"451b691f-2ec5-48a8-b54a-11da6c61fd64","Type":"ContainerStarted","Data":"119b8722f05052db20af98038c026a18d8ba05b4aa1ebe182f3734ae64b98c6c"}
Nov 01 06:23:11.884785 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:11.884598    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266" event={"ID":"451b691f-2ec5-48a8-b54a-11da6c61fd64","Type":"ContainerStarted","Data":"a279ba0253704794ed776e807c6a68d088e95b9f59cffdec15ce41f3ea085f4d"}
Nov 01 06:23:11.884785 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:11.884731    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266"
Nov 01 06:23:11.886301 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:11.886275    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz" event={"ID":"4cb4a057-3d46-441d-bb00-8057e63c9d37","Type":"ContainerStarted","Data":"1d8529fba4d4882c9a5e4c98b63c478a6ce2e837e46839983551c674309a4d2c"}
Nov 01 06:23:11.886446 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:11.886333    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz" event={"ID":"4cb4a057-3d46-441d-bb00-8057e63c9d37","Type":"ContainerStarted","Data":"4549b00ba98fe9a5b3c1484e05ebacd89cbd50fa3339d4d40517f8092ea15382"}
Nov 01 06:23:11.886446 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:11.886424    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz"
Nov 01 06:23:12.003104 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:12.003060    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266" podStartSLOduration=2.003047985 podStartE2EDuration="2.003047985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:23:12.002401019 +0000 UTC m=+2229.213224529" watchObservedRunningTime="2026-11-01 06:23:12.003047985 +0000 UTC m=+2229.213871494"
Nov 01 06:23:12.124083 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:12.119937    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz" podStartSLOduration=2.119917556 podStartE2EDuration="2.119917556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:23:12.119172224 +0000 UTC m=+2229.329995739" watchObservedRunningTime="2026-11-01 06:23:12.119917556 +0000 UTC m=+2229.330741066"
Nov 01 06:23:16.355737 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:16.355705    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:21.164050 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:21.164006    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:22.895680 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:22.895647    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266"
Nov 01 06:23:22.896082 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:22.895719    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz"
Nov 01 06:23:24.181095 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:24.181061    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:25.193046 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.193012    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"]
Nov 01 06:23:25.193468 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.193229    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk" podUID="7c0e5170-1714-4595-b348-1ee089572a62" containerName="maas-api" containerID="cri-o://ff6fc43e9cdc59c972e1bf299caef45fa6d3c76e6aa632529fa2fa137f9899dd" gracePeriod=30
Nov 01 06:23:25.294601 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.294566    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:25.479556 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.479529    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"
Nov 01 06:23:25.581772 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.580762    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c0e5170-1714-4595-b348-1ee089572a62-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c0e5170-1714-4595-b348-1ee089572a62-metrics-certs\") pod \"7c0e5170-1714-4595-b348-1ee089572a62\" (UID: \"7c0e5170-1714-4595-b348-1ee089572a62\") "
Nov 01 06:23:25.581772 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.580841    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c0e5170-1714-4595-b348-1ee089572a62-kube-api-access-mp9lt\" (UniqueName: \"kubernetes.io/projected/7c0e5170-1714-4595-b348-1ee089572a62-kube-api-access-mp9lt\") pod \"7c0e5170-1714-4595-b348-1ee089572a62\" (UID: \"7c0e5170-1714-4595-b348-1ee089572a62\") "
Nov 01 06:23:25.581772 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.581016    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c0e5170-1714-4595-b348-1ee089572a62-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c0e5170-1714-4595-b348-1ee089572a62-maas-api-tls\") pod \"7c0e5170-1714-4595-b348-1ee089572a62\" (UID: \"7c0e5170-1714-4595-b348-1ee089572a62\") "
Nov 01 06:23:25.581772 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.581085    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c0e5170-1714-4595-b348-1ee089572a62-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0e5170-1714-4595-b348-1ee089572a62-openshift-service-ca\") pod \"7c0e5170-1714-4595-b348-1ee089572a62\" (UID: \"7c0e5170-1714-4595-b348-1ee089572a62\") "
Nov 01 06:23:25.584290 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.583140    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c0e5170-1714-4595-b348-1ee089572a62-openshift-service-ca" pod "7c0e5170-1714-4595-b348-1ee089572a62" (UID: "7c0e5170-1714-4595-b348-1ee089572a62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:23:25.597822 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.597775    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c0e5170-1714-4595-b348-1ee089572a62-metrics-certs" pod "7c0e5170-1714-4595-b348-1ee089572a62" (UID: "7c0e5170-1714-4595-b348-1ee089572a62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:23:25.598818 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.598773    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c0e5170-1714-4595-b348-1ee089572a62-maas-api-tls" pod "7c0e5170-1714-4595-b348-1ee089572a62" (UID: "7c0e5170-1714-4595-b348-1ee089572a62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:23:25.601829 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.601805    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c0e5170-1714-4595-b348-1ee089572a62-kube-api-access-mp9lt" pod "7c0e5170-1714-4595-b348-1ee089572a62" (UID: "7c0e5170-1714-4595-b348-1ee089572a62"). InnerVolumeSpecName "kube-api-access-mp9lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:23:25.682255 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.682221    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c0e5170-1714-4595-b348-1ee089572a62-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:25.682255 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.682254    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp9lt\" (UniqueName: \"kubernetes.io/projected/7c0e5170-1714-4595-b348-1ee089572a62-kube-api-access-mp9lt\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:25.682469 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.682265    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c0e5170-1714-4595-b348-1ee089572a62-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:25.682469 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.682274    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0e5170-1714-4595-b348-1ee089572a62-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:25.880193 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:23:25.880154    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:23:25.971457 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.971423    5619 generic.go:356] "Generic (PLEG): container finished" podID="7c0e5170-1714-4595-b348-1ee089572a62" containerID="ff6fc43e9cdc59c972e1bf299caef45fa6d3c76e6aa632529fa2fa137f9899dd" exitCode=0
Nov 01 06:23:25.971617 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.971477    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"
Nov 01 06:23:25.971617 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.971508    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk" event={"ID":"7c0e5170-1714-4595-b348-1ee089572a62","Type":"ContainerDied","Data":"ff6fc43e9cdc59c972e1bf299caef45fa6d3c76e6aa632529fa2fa137f9899dd"}
Nov 01 06:23:25.971617 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.971567    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk" event={"ID":"7c0e5170-1714-4595-b348-1ee089572a62","Type":"ContainerDied","Data":"997fc717de7f8eec82e174ddb700683b0808184be3530e3aa343486408f10c51"}
Nov 01 06:23:25.971617 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.971596    5619 scope.go:122] "RemoveContainer" containerID="ff6fc43e9cdc59c972e1bf299caef45fa6d3c76e6aa632529fa2fa137f9899dd"
Nov 01 06:23:25.982390 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.982372    5619 scope.go:122] "RemoveContainer" containerID="ff6fc43e9cdc59c972e1bf299caef45fa6d3c76e6aa632529fa2fa137f9899dd"
Nov 01 06:23:25.982678 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:23:25.982657    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff6fc43e9cdc59c972e1bf299caef45fa6d3c76e6aa632529fa2fa137f9899dd\": container with ID starting with ff6fc43e9cdc59c972e1bf299caef45fa6d3c76e6aa632529fa2fa137f9899dd not found: ID does not exist" containerID="ff6fc43e9cdc59c972e1bf299caef45fa6d3c76e6aa632529fa2fa137f9899dd"
Nov 01 06:23:25.982718 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:25.982685    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff6fc43e9cdc59c972e1bf299caef45fa6d3c76e6aa632529fa2fa137f9899dd"} err="failed to get container status \"ff6fc43e9cdc59c972e1bf299caef45fa6d3c76e6aa632529fa2fa137f9899dd\": rpc error: code = NotFound desc = could not find container \"ff6fc43e9cdc59c972e1bf299caef45fa6d3c76e6aa632529fa2fa137f9899dd\": container with ID starting with ff6fc43e9cdc59c972e1bf299caef45fa6d3c76e6aa632529fa2fa137f9899dd not found: ID does not exist"
Nov 01 06:23:26.139414 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:26.139345    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"]
Nov 01 06:23:26.145534 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:26.145507    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e5d7100d-65d684957b-5qwkk"]
Nov 01 06:23:26.848887 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:26.848850    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:26.962150 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:26.962119    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c0e5170-1714-4595-b348-1ee089572a62" path="/var/lib/kubelet/pods/7c0e5170-1714-4595-b348-1ee089572a62/volumes"
Nov 01 06:23:27.071988 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:27.071949    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:27.227261 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:27.227174    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:27.472564 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:27.472532    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"]
Nov 01 06:23:27.472785 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:27.472760    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2" podUID="cb80df79-1ef0-4be1-8607-0394242a387a" containerName="istio-proxy" containerID="cri-o://66035ff2b9c73d6187591bcd22212cb257fac88e6bf595cbfb8c544ce65c1c43" gracePeriod=30
Nov 01 06:23:27.597356 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:27.597233    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:28.077310 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:28.077228    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:28.979279 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:28.979243    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:29.497835 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.497803    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"]
Nov 01 06:23:29.498674 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.498652    5619 cpu_manager.go:414] "containerMap: removing container" podUID="7c0e5170-1714-4595-b348-1ee089572a62" containerName="maas-api"
Nov 01 06:23:29.498824 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.498676    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c0e5170-1714-4595-b348-1ee089572a62" containerName="maas-api"
Nov 01 06:23:29.498876 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.498827    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c0e5170-1714-4595-b348-1ee089572a62" containerName="maas-api"
Nov 01 06:23:29.504073 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.504053    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.533012 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.532980    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-podinfo\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.533129 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.533041    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-workload-certs\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.533183 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.533117    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-envoy\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.533183 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.533146    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istiod-ca-cert\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.533183 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.533174    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf2f8\" (UniqueName: \"kubernetes.io/projected/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-kube-api-access-xf2f8\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.533412 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.533388    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-data\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.533548 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.533528    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-token\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.533677 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.533655    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-credential-socket\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.533724 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.533709    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-workload-socket\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.557593 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.557566    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-83558e40-openshift-default-dockercfg-fdgtd\""
Nov 01 06:23:29.596787 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.596749    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:29.610898 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.610869    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"]
Nov 01 06:23:29.634705 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.634672    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-token\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.634872 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.634740    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-credential-socket\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.634872 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.634767    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-workload-socket\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.634872 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.634798    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-podinfo\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.634872 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.634829    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-workload-certs\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.634872 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.634853    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-envoy\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.635236 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.634876    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istiod-ca-cert\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.635236 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.634905    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf2f8\" (UniqueName: \"kubernetes.io/projected/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-kube-api-access-xf2f8\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.635236 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.635029    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-data\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.635500 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.635473    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-data\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.635688 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.635617    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-credential-socket\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.635793 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.635760    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-workload-certs\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.635890 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.635869    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istiod-ca-cert\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.635952 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.635922    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-workload-socket\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.637730 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.637706    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-envoy\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.638387 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.638361    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-podinfo\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.697104 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.697067    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-token\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.697264 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.697236    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf2f8\" (UniqueName: \"kubernetes.io/projected/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-kube-api-access-xf2f8\") pod \"e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") " pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.817351 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.817250    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:29.901986 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:29.897889    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:30.260073 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:30.260047    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"]
Nov 01 06:23:30.261130 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:23:30.261104    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c0ea9c5_f467_4031_8bd3_8a724d3848e9.slice/crio-6c149dc793fbddd1a79934414379b82999dc49d0bfd75db21501a4809a84aae7 WatchSource:0}: Error finding container 6c149dc793fbddd1a79934414379b82999dc49d0bfd75db21501a4809a84aae7: Status 404 returned error can't find the container with id 6c149dc793fbddd1a79934414379b82999dc49d0bfd75db21501a4809a84aae7
Nov 01 06:23:30.263697 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:30.263666    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:23:30.263775 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:30.263735    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:23:30.263814 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:30.263771    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:23:31.004069 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:31.004030    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7" event={"ID":"4c0ea9c5-f467-4031-8bd3-8a724d3848e9","Type":"ContainerStarted","Data":"ea2804a86cf5b3abc43a00fbd348ac97197ae12b0ca5c47b7034912cd9775904"}
Nov 01 06:23:31.004487 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:31.004082    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7" event={"ID":"4c0ea9c5-f467-4031-8bd3-8a724d3848e9","Type":"ContainerStarted","Data":"6c149dc793fbddd1a79934414379b82999dc49d0bfd75db21501a4809a84aae7"}
Nov 01 06:23:31.127889 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:31.127836    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7" podStartSLOduration=2.127821716 podStartE2EDuration="2.127821716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:23:31.125517107 +0000 UTC m=+2248.336340611" watchObservedRunningTime="2026-11-01 06:23:31.127821716 +0000 UTC m=+2248.338645225"
Nov 01 06:23:31.264983 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:31.264892    5619 patch_prober.go:29] interesting pod/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:23:31.264983 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:31.264959    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2" podUID="cb80df79-1ef0-4be1-8607-0394242a387a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:23:31.818176 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:31.818144    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:31.823907 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:31.823883    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:32.009959 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.009925    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:32.010844 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.010826    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:23:32.133804 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.133770    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:32.738967 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.738910    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:23:32.774156 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.774130    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-istio-data\") pod \"cb80df79-1ef0-4be1-8607-0394242a387a\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") "
Nov 01 06:23:32.774281 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.774162    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-workload-socket\") pod \"cb80df79-1ef0-4be1-8607-0394242a387a\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") "
Nov 01 06:23:32.774281 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.774240    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-workload-certs\") pod \"cb80df79-1ef0-4be1-8607-0394242a387a\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") "
Nov 01 06:23:32.774281 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.774262    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb80df79-1ef0-4be1-8607-0394242a387a-istio-token\" (UniqueName: \"kubernetes.io/projected/cb80df79-1ef0-4be1-8607-0394242a387a-istio-token\") pod \"cb80df79-1ef0-4be1-8607-0394242a387a\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") "
Nov 01 06:23:32.774489 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.774293    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb80df79-1ef0-4be1-8607-0394242a387a-kube-api-access-vk4jm\" (UniqueName: \"kubernetes.io/projected/cb80df79-1ef0-4be1-8607-0394242a387a-kube-api-access-vk4jm\") pod \"cb80df79-1ef0-4be1-8607-0394242a387a\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") "
Nov 01 06:23:32.774489 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.774443    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-istio-data" pod "cb80df79-1ef0-4be1-8607-0394242a387a" (UID: "cb80df79-1ef0-4be1-8607-0394242a387a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:23:32.774489 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.774477    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb80df79-1ef0-4be1-8607-0394242a387a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb80df79-1ef0-4be1-8607-0394242a387a-istiod-ca-cert\") pod \"cb80df79-1ef0-4be1-8607-0394242a387a\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") "
Nov 01 06:23:32.774638 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.774518    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-workload-socket" pod "cb80df79-1ef0-4be1-8607-0394242a387a" (UID: "cb80df79-1ef0-4be1-8607-0394242a387a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:23:32.774638 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.774525    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-istio-envoy\") pod \"cb80df79-1ef0-4be1-8607-0394242a387a\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") "
Nov 01 06:23:32.774638 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.774564    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-workload-certs" pod "cb80df79-1ef0-4be1-8607-0394242a387a" (UID: "cb80df79-1ef0-4be1-8607-0394242a387a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:23:32.774638 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.774615    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb80df79-1ef0-4be1-8607-0394242a387a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb80df79-1ef0-4be1-8607-0394242a387a-istio-podinfo\") pod \"cb80df79-1ef0-4be1-8607-0394242a387a\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") "
Nov 01 06:23:32.774845 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.774670    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-credential-socket\") pod \"cb80df79-1ef0-4be1-8607-0394242a387a\" (UID: \"cb80df79-1ef0-4be1-8607-0394242a387a\") "
Nov 01 06:23:32.774845 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.774837    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb80df79-1ef0-4be1-8607-0394242a387a-istiod-ca-cert" pod "cb80df79-1ef0-4be1-8607-0394242a387a" (UID: "cb80df79-1ef0-4be1-8607-0394242a387a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:23:32.775045 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.775019    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-credential-socket" pod "cb80df79-1ef0-4be1-8607-0394242a387a" (UID: "cb80df79-1ef0-4be1-8607-0394242a387a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:23:32.775279 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.775252    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:32.775279 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.775277    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:32.775466 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.775288    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:32.775466 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.775299    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:32.775466 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.775308    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb80df79-1ef0-4be1-8607-0394242a387a-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:32.776731 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.776701    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb80df79-1ef0-4be1-8607-0394242a387a-istio-podinfo" pod "cb80df79-1ef0-4be1-8607-0394242a387a" (UID: "cb80df79-1ef0-4be1-8607-0394242a387a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:23:32.777033 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.777000    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-istio-envoy" pod "cb80df79-1ef0-4be1-8607-0394242a387a" (UID: "cb80df79-1ef0-4be1-8607-0394242a387a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:23:32.777154 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.777129    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb80df79-1ef0-4be1-8607-0394242a387a-istio-token" pod "cb80df79-1ef0-4be1-8607-0394242a387a" (UID: "cb80df79-1ef0-4be1-8607-0394242a387a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:23:32.777220 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.777157    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb80df79-1ef0-4be1-8607-0394242a387a-kube-api-access-vk4jm" pod "cb80df79-1ef0-4be1-8607-0394242a387a" (UID: "cb80df79-1ef0-4be1-8607-0394242a387a"). InnerVolumeSpecName "kube-api-access-vk4jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:23:32.876864 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.876837    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb80df79-1ef0-4be1-8607-0394242a387a-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:32.876864 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.876863    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb80df79-1ef0-4be1-8607-0394242a387a-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:32.877047 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.876872    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb80df79-1ef0-4be1-8607-0394242a387a-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:32.877047 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:32.876882    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk4jm\" (UniqueName: \"kubernetes.io/projected/cb80df79-1ef0-4be1-8607-0394242a387a-kube-api-access-vk4jm\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:23:33.017495 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:33.017425    5619 generic.go:356] "Generic (PLEG): container finished" podID="cb80df79-1ef0-4be1-8607-0394242a387a" containerID="66035ff2b9c73d6187591bcd22212cb257fac88e6bf595cbfb8c544ce65c1c43" exitCode=0
Nov 01 06:23:33.017862 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:33.017492    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"
Nov 01 06:23:33.017862 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:33.017513    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2" event={"ID":"cb80df79-1ef0-4be1-8607-0394242a387a","Type":"ContainerDied","Data":"66035ff2b9c73d6187591bcd22212cb257fac88e6bf595cbfb8c544ce65c1c43"}
Nov 01 06:23:33.017862 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:33.017570    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2" event={"ID":"cb80df79-1ef0-4be1-8607-0394242a387a","Type":"ContainerDied","Data":"54f59daeb4ad6833ef5f0f1e760e0e08856b11f017edc42f3f5ee1d2cc9de644"}
Nov 01 06:23:33.017862 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:33.017601    5619 scope.go:122] "RemoveContainer" containerID="66035ff2b9c73d6187591bcd22212cb257fac88e6bf595cbfb8c544ce65c1c43"
Nov 01 06:23:33.026909 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:33.026888    5619 scope.go:122] "RemoveContainer" containerID="b198e622e33cb9aa30a4246239a00504ab2f3c26cc77339c73a3b94e60995126"
Nov 01 06:23:33.035290 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:33.035273    5619 scope.go:122] "RemoveContainer" containerID="66035ff2b9c73d6187591bcd22212cb257fac88e6bf595cbfb8c544ce65c1c43"
Nov 01 06:23:33.035608 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:23:33.035582    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66035ff2b9c73d6187591bcd22212cb257fac88e6bf595cbfb8c544ce65c1c43\": container with ID starting with 66035ff2b9c73d6187591bcd22212cb257fac88e6bf595cbfb8c544ce65c1c43 not found: ID does not exist" containerID="66035ff2b9c73d6187591bcd22212cb257fac88e6bf595cbfb8c544ce65c1c43"
Nov 01 06:23:33.035714 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:33.035621    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66035ff2b9c73d6187591bcd22212cb257fac88e6bf595cbfb8c544ce65c1c43"} err="failed to get container status \"66035ff2b9c73d6187591bcd22212cb257fac88e6bf595cbfb8c544ce65c1c43\": rpc error: code = NotFound desc = could not find container \"66035ff2b9c73d6187591bcd22212cb257fac88e6bf595cbfb8c544ce65c1c43\": container with ID starting with 66035ff2b9c73d6187591bcd22212cb257fac88e6bf595cbfb8c544ce65c1c43 not found: ID does not exist"
Nov 01 06:23:33.035714 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:33.035644    5619 scope.go:122] "RemoveContainer" containerID="b198e622e33cb9aa30a4246239a00504ab2f3c26cc77339c73a3b94e60995126"
Nov 01 06:23:33.035897 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:23:33.035879    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b198e622e33cb9aa30a4246239a00504ab2f3c26cc77339c73a3b94e60995126\": container with ID starting with b198e622e33cb9aa30a4246239a00504ab2f3c26cc77339c73a3b94e60995126 not found: ID does not exist" containerID="b198e622e33cb9aa30a4246239a00504ab2f3c26cc77339c73a3b94e60995126"
Nov 01 06:23:33.035950 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:33.035903    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b198e622e33cb9aa30a4246239a00504ab2f3c26cc77339c73a3b94e60995126"} err="failed to get container status \"b198e622e33cb9aa30a4246239a00504ab2f3c26cc77339c73a3b94e60995126\": rpc error: code = NotFound desc = could not find container \"b198e622e33cb9aa30a4246239a00504ab2f3c26cc77339c73a3b94e60995126\": container with ID starting with b198e622e33cb9aa30a4246239a00504ab2f3c26cc77339c73a3b94e60995126 not found: ID does not exist"
Nov 01 06:23:33.185043 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:33.185013    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"]
Nov 01 06:23:33.186535 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:33.186506    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:33.189492 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:33.189469    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e5d7100d-gw-openshift-default-5769d95589-xf8j2"]
Nov 01 06:23:33.346859 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:33.346813    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:33.493515 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:33.493476    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:34.964944 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:34.964905    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb80df79-1ef0-4be1-8607-0394242a387a" path="/var/lib/kubelet/pods/cb80df79-1ef0-4be1-8607-0394242a387a/volumes"
Nov 01 06:23:36.378661 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:36.378618    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:36.555055 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:36.555020    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:36.796763 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:36.796685    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6"]
Nov 01 06:23:36.797572 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:36.797547    5619 cpu_manager.go:414] "containerMap: removing container" podUID="cb80df79-1ef0-4be1-8607-0394242a387a" containerName="istio-proxy"
Nov 01 06:23:36.797572 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:36.797569    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb80df79-1ef0-4be1-8607-0394242a387a" containerName="istio-proxy"
Nov 01 06:23:36.797751 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:36.797719    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb80df79-1ef0-4be1-8607-0394242a387a" containerName="istio-proxy"
Nov 01 06:23:36.797751 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:36.797734    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb80df79-1ef0-4be1-8607-0394242a387a" containerName="istio-proxy"
Nov 01 06:23:36.802911 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:36.802889    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6"
Nov 01 06:23:36.938373 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:36.935287    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-83558e40-dockercfg-qqr7z\""
Nov 01 06:23:36.940479 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:36.940444    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6"]
Nov 01 06:23:36.940854 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:36.940832    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-83558e40\""
Nov 01 06:23:36.942313 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:36.941874    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ls5z\" (UniqueName: \"kubernetes.io/projected/935d319f-3873-4cb4-b654-bd4dbb8221a3-kube-api-access-5ls5z\") pod \"payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6\" (UID: \"935d319f-3873-4cb4-b654-bd4dbb8221a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6"
Nov 01 06:23:36.942645 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:36.942620    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935d319f-3873-4cb4-b654-bd4dbb8221a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6\" (UID: \"935d319f-3873-4cb4-b654-bd4dbb8221a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6"
Nov 01 06:23:36.948189 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:36.948153    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc"]
Nov 01 06:23:36.949268 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:36.949243    5619 cpu_manager.go:414] "containerMap: removing container" podUID="cb80df79-1ef0-4be1-8607-0394242a387a" containerName="istio-proxy"
Nov 01 06:23:36.949423 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:36.949406    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb80df79-1ef0-4be1-8607-0394242a387a" containerName="istio-proxy"
Nov 01 06:23:36.964703 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:36.964677    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc"
Nov 01 06:23:37.016866 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:37.016835    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc"]
Nov 01 06:23:37.044266 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:37.044222    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0b83090-6be0-4931-8695-9e313298fefe-plugins-config-volume\") pod \"payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc\" (UID: \"a0b83090-6be0-4931-8695-9e313298fefe\") " pod="openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc"
Nov 01 06:23:37.044466 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:37.044418    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tbfb\" (UniqueName: \"kubernetes.io/projected/a0b83090-6be0-4931-8695-9e313298fefe-kube-api-access-9tbfb\") pod \"payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc\" (UID: \"a0b83090-6be0-4931-8695-9e313298fefe\") " pod="openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc"
Nov 01 06:23:37.044607 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:37.044585    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ls5z\" (UniqueName: \"kubernetes.io/projected/935d319f-3873-4cb4-b654-bd4dbb8221a3-kube-api-access-5ls5z\") pod \"payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6\" (UID: \"935d319f-3873-4cb4-b654-bd4dbb8221a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6"
Nov 01 06:23:37.044716 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:37.044682    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935d319f-3873-4cb4-b654-bd4dbb8221a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6\" (UID: \"935d319f-3873-4cb4-b654-bd4dbb8221a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6"
Nov 01 06:23:37.045529 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:37.045507    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935d319f-3873-4cb4-b654-bd4dbb8221a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6\" (UID: \"935d319f-3873-4cb4-b654-bd4dbb8221a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6"
Nov 01 06:23:37.106815 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:37.106781    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ls5z\" (UniqueName: \"kubernetes.io/projected/935d319f-3873-4cb4-b654-bd4dbb8221a3-kube-api-access-5ls5z\") pod \"payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6\" (UID: \"935d319f-3873-4cb4-b654-bd4dbb8221a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6"
Nov 01 06:23:37.119842 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:37.119817    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6"
Nov 01 06:23:37.145824 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:37.145796    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0b83090-6be0-4931-8695-9e313298fefe-plugins-config-volume\") pod \"payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc\" (UID: \"a0b83090-6be0-4931-8695-9e313298fefe\") " pod="openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc"
Nov 01 06:23:37.145946 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:37.145880    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tbfb\" (UniqueName: \"kubernetes.io/projected/a0b83090-6be0-4931-8695-9e313298fefe-kube-api-access-9tbfb\") pod \"payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc\" (UID: \"a0b83090-6be0-4931-8695-9e313298fefe\") " pod="openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc"
Nov 01 06:23:37.146551 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:37.146532    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0b83090-6be0-4931-8695-9e313298fefe-plugins-config-volume\") pod \"payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc\" (UID: \"a0b83090-6be0-4931-8695-9e313298fefe\") " pod="openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc"
Nov 01 06:23:37.207842 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:37.207815    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tbfb\" (UniqueName: \"kubernetes.io/projected/a0b83090-6be0-4931-8695-9e313298fefe-kube-api-access-9tbfb\") pod \"payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc\" (UID: \"a0b83090-6be0-4931-8695-9e313298fefe\") " pod="openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc"
Nov 01 06:23:37.289142 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:37.289102    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc"
Nov 01 06:23:37.487146 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:37.487060    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:37.507074 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:37.507053    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6"]
Nov 01 06:23:37.508473 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:23:37.508448    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935d319f_3873_4cb4_b654_bd4dbb8221a3.slice/crio-6f43c24729ef88c1ff6b99758e66001a6afdae89f7dac3fb59b1cf70144d4fb4 WatchSource:0}: Error finding container 6f43c24729ef88c1ff6b99758e66001a6afdae89f7dac3fb59b1cf70144d4fb4: Status 404 returned error can't find the container with id 6f43c24729ef88c1ff6b99758e66001a6afdae89f7dac3fb59b1cf70144d4fb4
Nov 01 06:23:37.673985 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:37.673955    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc"]
Nov 01 06:23:37.675431 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:23:37.675401    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b83090_6be0_4931_8695_9e313298fefe.slice/crio-9258100b6889dda4c7e211335c7d69949f7d4a42525b7702c53f8f19b0bd33b9 WatchSource:0}: Error finding container 9258100b6889dda4c7e211335c7d69949f7d4a42525b7702c53f8f19b0bd33b9: Status 404 returned error can't find the container with id 9258100b6889dda4c7e211335c7d69949f7d4a42525b7702c53f8f19b0bd33b9
Nov 01 06:23:38.053941 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:38.053896    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6" event={"ID":"935d319f-3873-4cb4-b654-bd4dbb8221a3","Type":"ContainerStarted","Data":"85fd7a6f49d03af8f6a0df850bd34eba49decbf15e9174dfe7cb27490eb6449e"}
Nov 01 06:23:38.054130 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:38.053971    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6" event={"ID":"935d319f-3873-4cb4-b654-bd4dbb8221a3","Type":"ContainerStarted","Data":"6f43c24729ef88c1ff6b99758e66001a6afdae89f7dac3fb59b1cf70144d4fb4"}
Nov 01 06:23:38.054201 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:38.054182    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6"
Nov 01 06:23:38.055624 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:38.055596    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc" event={"ID":"a0b83090-6be0-4931-8695-9e313298fefe","Type":"ContainerStarted","Data":"a8ef238bc2f1740b5742d70c235dc85d57cdba5a50bb4d15c67ee34deca78057"}
Nov 01 06:23:38.055769 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:38.055638    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc" event={"ID":"a0b83090-6be0-4931-8695-9e313298fefe","Type":"ContainerStarted","Data":"9258100b6889dda4c7e211335c7d69949f7d4a42525b7702c53f8f19b0bd33b9"}
Nov 01 06:23:38.055769 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:38.055672    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc"
Nov 01 06:23:38.172026 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:38.171969    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6" podStartSLOduration=2.1719521569999998 podStartE2EDuration="2.171952157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:23:38.170974041 +0000 UTC m=+2255.381797551" watchObservedRunningTime="2026-11-01 06:23:38.171952157 +0000 UTC m=+2255.382775667"
Nov 01 06:23:38.290372 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:38.290290    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc" podStartSLOduration=2.290269756 podStartE2EDuration="2.290269756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:23:38.288916526 +0000 UTC m=+2255.499740035" watchObservedRunningTime="2026-11-01 06:23:38.290269756 +0000 UTC m=+2255.501093266"
Nov 01 06:23:42.196588 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:42.196487    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:44.927722 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:44.927688    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:45.777585 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:45.777551    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:46.974624 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:46.974586    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:47.899928 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:47.899887    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:49.064921 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:49.064891    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6"
Nov 01 06:23:49.065416 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:49.064958    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc"
Nov 01 06:23:51.758300 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:51.758258    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:53.488933 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:53.488896    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:53.686589 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:53.686549    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:54.118891 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:54.118853    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:55.879444 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:23:55.879409    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:23:56.231128 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:56.231053    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"]
Nov 01 06:23:56.234823 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:56.234806    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"
Nov 01 06:23:56.278725 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:56.278702    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d452a9ec-fe57-41b2-be6f-26ed3a942f32-metrics-certs\") pod \"maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28\" (UID: \"d452a9ec-fe57-41b2-be6f-26ed3a942f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"
Nov 01 06:23:56.278873 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:56.278751    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmd9p\" (UniqueName: \"kubernetes.io/projected/d452a9ec-fe57-41b2-be6f-26ed3a942f32-kube-api-access-vmd9p\") pod \"maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28\" (UID: \"d452a9ec-fe57-41b2-be6f-26ed3a942f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"
Nov 01 06:23:56.278873 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:56.278779    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d452a9ec-fe57-41b2-be6f-26ed3a942f32-openshift-service-ca\") pod \"maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28\" (UID: \"d452a9ec-fe57-41b2-be6f-26ed3a942f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"
Nov 01 06:23:56.278999 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:56.278886    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d452a9ec-fe57-41b2-be6f-26ed3a942f32-maas-api-tls\") pod \"maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28\" (UID: \"d452a9ec-fe57-41b2-be6f-26ed3a942f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"
Nov 01 06:23:56.288162 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:56.288118    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c7e551a7-serving-cert\""
Nov 01 06:23:56.341773 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:56.341740    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"]
Nov 01 06:23:56.379714 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:56.379675    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d452a9ec-fe57-41b2-be6f-26ed3a942f32-maas-api-tls\") pod \"maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28\" (UID: \"d452a9ec-fe57-41b2-be6f-26ed3a942f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"
Nov 01 06:23:56.379875 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:56.379856    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d452a9ec-fe57-41b2-be6f-26ed3a942f32-metrics-certs\") pod \"maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28\" (UID: \"d452a9ec-fe57-41b2-be6f-26ed3a942f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"
Nov 01 06:23:56.379940 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:23:56.379876    5619 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-serving-cert: secret "maas-api-e2e-mt-c7e551a7-serving-cert" not found
Nov 01 06:23:56.379940 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:56.379929    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmd9p\" (UniqueName: \"kubernetes.io/projected/d452a9ec-fe57-41b2-be6f-26ed3a942f32-kube-api-access-vmd9p\") pod \"maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28\" (UID: \"d452a9ec-fe57-41b2-be6f-26ed3a942f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"
Nov 01 06:23:56.380048 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:23:56.379946    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d452a9ec-fe57-41b2-be6f-26ed3a942f32-maas-api-tls podName:d452a9ec-fe57-41b2-be6f-26ed3a942f32 nodeName:}" failed. No retries permitted until 2026-11-01 06:23:56.879923 +0000 UTC m=+2274.090746501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d452a9ec-fe57-41b2-be6f-26ed3a942f32-maas-api-tls") pod "maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28" (UID: "d452a9ec-fe57-41b2-be6f-26ed3a942f32") : secret "maas-api-e2e-mt-c7e551a7-serving-cert" not found
Nov 01 06:23:56.380048 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:56.380001    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d452a9ec-fe57-41b2-be6f-26ed3a942f32-openshift-service-ca\") pod \"maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28\" (UID: \"d452a9ec-fe57-41b2-be6f-26ed3a942f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"
Nov 01 06:23:56.380812 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:56.380785    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d452a9ec-fe57-41b2-be6f-26ed3a942f32-openshift-service-ca\") pod \"maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28\" (UID: \"d452a9ec-fe57-41b2-be6f-26ed3a942f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"
Nov 01 06:23:56.382878 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:56.382818    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d452a9ec-fe57-41b2-be6f-26ed3a942f32-metrics-certs\") pod \"maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28\" (UID: \"d452a9ec-fe57-41b2-be6f-26ed3a942f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"
Nov 01 06:23:56.430656 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:56.430623    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:56.442062 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:56.442039    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmd9p\" (UniqueName: \"kubernetes.io/projected/d452a9ec-fe57-41b2-be6f-26ed3a942f32-kube-api-access-vmd9p\") pod \"maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28\" (UID: \"d452a9ec-fe57-41b2-be6f-26ed3a942f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"
Nov 01 06:23:56.627256 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:56.627209    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:56.773422 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:56.773387    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:23:56.887706 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:56.887618    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d452a9ec-fe57-41b2-be6f-26ed3a942f32-maas-api-tls\") pod \"maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28\" (UID: \"d452a9ec-fe57-41b2-be6f-26ed3a942f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"
Nov 01 06:23:56.889787 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:56.889765    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d452a9ec-fe57-41b2-be6f-26ed3a942f32-maas-api-tls\") pod \"maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28\" (UID: \"d452a9ec-fe57-41b2-be6f-26ed3a942f32\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"
Nov 01 06:23:57.149004 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:57.148920    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"
Nov 01 06:23:57.524674 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:57.524643    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"]
Nov 01 06:23:57.525705 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:23:57.525679    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd452a9ec_fe57_41b2_be6f_26ed3a942f32.slice/crio-4b0607c6866ecaf71addbd6b6037150852b56c763bf3104744b8047a7d18c16d WatchSource:0}: Error finding container 4b0607c6866ecaf71addbd6b6037150852b56c763bf3104744b8047a7d18c16d: Status 404 returned error can't find the container with id 4b0607c6866ecaf71addbd6b6037150852b56c763bf3104744b8047a7d18c16d
Nov 01 06:23:57.527587 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:57.527563    5619 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 06:23:58.183624 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:58.183578    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28" event={"ID":"d452a9ec-fe57-41b2-be6f-26ed3a942f32","Type":"ContainerStarted","Data":"4b0607c6866ecaf71addbd6b6037150852b56c763bf3104744b8047a7d18c16d"}
Nov 01 06:23:59.192509 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:59.192467    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28" event={"ID":"d452a9ec-fe57-41b2-be6f-26ed3a942f32","Type":"ContainerStarted","Data":"8400c396af40d27bca36da9217ae032cf47ea47f46e95a02e1d8f306de487342"}
Nov 01 06:23:59.316687 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:23:59.316629    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28" podStartSLOduration=2.377690395 podStartE2EDuration="3.316614076s" totalImagesPullingTime="938.923681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:23:59.314284923 +0000 UTC m=+2276.525108432" watchObservedRunningTime="2026-11-01 06:23:59.316614076 +0000 UTC m=+2276.527437585"
Nov 01 06:24:00.018823 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:00.018786    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:00.200048 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:00.200014    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"
Nov 01 06:24:00.211203 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:00.211166    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:01.387003 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:01.386969    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:06.164739 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:06.164696    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:06.213515 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:06.213473    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"
Nov 01 06:24:10.756083 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:10.756051    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:10.900013 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:10.899964    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:11.609061 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:11.609030    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:15.666618 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:15.666573    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:20.459877 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:20.459836    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:24.535302 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:24.535262    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:25.202278 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:25.202236    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:25.362495 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:25.362459    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:25.731809 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:25.731776    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:25.933054 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:24:25.933011    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:24:27.684978 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:27.684937    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:29.355186 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:29.355146    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:30.180994 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:30.180960    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:34.360000 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:34.359963    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:34.506431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:34.506383    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:34.755606 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:34.755510    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:34.974011 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:34.973967    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:36.594945 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:36.594911    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:37.439704 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:37.439667    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:39.758549 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:39.758507    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:41.125063 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:41.125029    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:41.747293 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:41.747258    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:41.994971 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:41.994931    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:44.585552 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:44.585518    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:49.412019 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:49.411983    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:51.159029 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:51.158936    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:54.182510 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:54.182421    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:54.551140 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:54.551105    5619 generic.go:356] "Generic (PLEG): container finished" podID="4c0ea9c5-f467-4031-8bd3-8a724d3848e9" containerID="ea2804a86cf5b3abc43a00fbd348ac97197ae12b0ca5c47b7034912cd9775904" exitCode=0
Nov 01 06:24:54.551361 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:54.551180    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7" event={"ID":"4c0ea9c5-f467-4031-8bd3-8a724d3848e9","Type":"ContainerDied","Data":"ea2804a86cf5b3abc43a00fbd348ac97197ae12b0ca5c47b7034912cd9775904"}
Nov 01 06:24:54.551671 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:54.551653    5619 scope.go:122] "RemoveContainer" containerID="ea2804a86cf5b3abc43a00fbd348ac97197ae12b0ca5c47b7034912cd9775904"
Nov 01 06:24:54.553498 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:54.553464    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:24:54.553587 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:54.553547    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:24:54.553630 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:54.553613    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:24:55.580748 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:55.580714    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7" event={"ID":"4c0ea9c5-f467-4031-8bd3-8a724d3848e9","Type":"ContainerStarted","Data":"6c32baee4921ca84080a606fc8d81ebed89fa61c372406828b2c1018fec3da20"}
Nov 01 06:24:55.817887 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:55.817803    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:24:55.818042 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:55.817963    5619 patch_prober.go:29] interesting pod/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 01 06:24:55.818042 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:55.818018    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7" podUID="4c0ea9c5-f467-4031-8bd3-8a724d3848e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 01 06:24:55.901111 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:24:55.901075    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:24:56.818540 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:56.818511    5619 patch_prober.go:29] interesting pod/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 01 06:24:56.818899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:56.818575    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7" podUID="4c0ea9c5-f467-4031-8bd3-8a724d3848e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 01 06:24:57.818166 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:57.818122    5619 patch_prober.go:29] interesting pod/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 01 06:24:57.818391 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:57.818191    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7" podUID="4c0ea9c5-f467-4031-8bd3-8a724d3848e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 01 06:24:57.831034 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:57.830989    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:57.980527 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:57.980495    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:58.270347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:58.268895    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:58.817662 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:58.817623    5619 patch_prober.go:29] interesting pod/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 01 06:24:58.817847 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:58.817698    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7" podUID="4c0ea9c5-f467-4031-8bd3-8a724d3848e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 01 06:24:59.098731 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:59.098689    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:24:59.817663 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:59.817631    5619 patch_prober.go:29] interesting pod/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 01 06:24:59.817845 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:24:59.817688    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7" podUID="4c0ea9c5-f467-4031-8bd3-8a724d3848e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 01 06:25:00.822040 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:00.822011    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:25:00.822552 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:00.822350    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:25:00.823113 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:00.823094    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:25:04.082754 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:04.082721    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:04.271810 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:04.271779    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:04.422893 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:04.422855    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:04.596904 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:04.596857    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:04.984393 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:04.984359    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:05.133753 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:05.133718    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:05.999002 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:05.998965    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3986bf/agc-cred-6204f08c208e: secret "agc-cred-6204f08c208e" not found
Nov 01 06:25:05.999002 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:05.999003    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4: secret "agc-cred-6204f08c208e" not found
Nov 01 06:25:05.999365 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:05.999076    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-provider-credentials podName:5ec32d57-998e-4646-885b-094f7fbaafd7 nodeName:}" failed. No retries permitted until 2026-11-01 06:25:06.499051543 +0000 UTC m=+2343.709875043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4" (UID: "5ec32d57-998e-4646-885b-094f7fbaafd7") : secret "agc-cred-6204f08c208e" not found
Nov 01 06:25:06.504818 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:06.504783    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3986bf/agc-cred-6204f08c208e: secret "agc-cred-6204f08c208e" not found
Nov 01 06:25:06.504818 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:06.504813    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4: secret "agc-cred-6204f08c208e" not found
Nov 01 06:25:06.505242 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:06.504877    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-provider-credentials podName:5ec32d57-998e-4646-885b-094f7fbaafd7 nodeName:}" failed. No retries permitted until 2026-11-01 06:25:07.504858334 +0000 UTC m=+2344.715681837 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4" (UID: "5ec32d57-998e-4646-885b-094f7fbaafd7") : secret "agc-cred-6204f08c208e" not found
Nov 01 06:25:07.519744 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:07.519707    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3986bf/agc-cred-6204f08c208e: secret "agc-cred-6204f08c208e" not found
Nov 01 06:25:07.519744 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:07.519739    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4: secret "agc-cred-6204f08c208e" not found
Nov 01 06:25:07.520150 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:07.519794    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-provider-credentials podName:5ec32d57-998e-4646-885b-094f7fbaafd7 nodeName:}" failed. No retries permitted until 2026-11-01 06:25:09.519779602 +0000 UTC m=+2346.730603108 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4" (UID: "5ec32d57-998e-4646-885b-094f7fbaafd7") : secret "agc-cred-6204f08c208e" not found
Nov 01 06:25:09.118067 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:09.118037    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:09.290247 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:09.290213    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:09.553537 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:09.553460    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3986bf/agc-cred-6204f08c208e: secret "agc-cred-6204f08c208e" not found
Nov 01 06:25:09.553537 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:09.553494    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4: secret "agc-cred-6204f08c208e" not found
Nov 01 06:25:09.553720 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:09.553565    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-provider-credentials podName:5ec32d57-998e-4646-885b-094f7fbaafd7 nodeName:}" failed. No retries permitted until 2026-11-01 06:25:13.553545932 +0000 UTC m=+2350.764369442 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4" (UID: "5ec32d57-998e-4646-885b-094f7fbaafd7") : secret "agc-cred-6204f08c208e" not found
Nov 01 06:25:10.927606 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:10.927523    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:11.101499 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.101463    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:11.345662 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.345623    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g"]
Nov 01 06:25:11.349831 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.349804    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g"
Nov 01 06:25:11.465876 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.465841    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f13905-dockercfg-xpq5p\""
Nov 01 06:25:11.480103 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.480071    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f13905\""
Nov 01 06:25:11.504126 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.489215    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g"]
Nov 01 06:25:11.504469 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.504450    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5"]
Nov 01 06:25:11.505503 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.501467    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvfnq\" (UniqueName: \"kubernetes.io/projected/5cbf3386-2dbf-4f96-8f0a-a7d90fe862df-kube-api-access-nvfnq\") pod \"payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g\" (UID: \"5cbf3386-2dbf-4f96-8f0a-a7d90fe862df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g"
Nov 01 06:25:11.505679 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.505579    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cbf3386-2dbf-4f96-8f0a-a7d90fe862df-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g\" (UID: \"5cbf3386-2dbf-4f96-8f0a-a7d90fe862df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g"
Nov 01 06:25:11.511672 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.511649    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5"
Nov 01 06:25:11.564057 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.564029    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5"]
Nov 01 06:25:11.606914 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.606821    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvfnq\" (UniqueName: \"kubernetes.io/projected/5cbf3386-2dbf-4f96-8f0a-a7d90fe862df-kube-api-access-nvfnq\") pod \"payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g\" (UID: \"5cbf3386-2dbf-4f96-8f0a-a7d90fe862df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g"
Nov 01 06:25:11.607057 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.606975    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cbf3386-2dbf-4f96-8f0a-a7d90fe862df-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g\" (UID: \"5cbf3386-2dbf-4f96-8f0a-a7d90fe862df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g"
Nov 01 06:25:11.607057 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.607044    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjt8f\" (UniqueName: \"kubernetes.io/projected/4c806c76-debc-4f3d-88dc-da5ec7b978b0-kube-api-access-wjt8f\") pod \"payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5\" (UID: \"4c806c76-debc-4f3d-88dc-da5ec7b978b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5"
Nov 01 06:25:11.607176 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.607162    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c806c76-debc-4f3d-88dc-da5ec7b978b0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5\" (UID: \"4c806c76-debc-4f3d-88dc-da5ec7b978b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5"
Nov 01 06:25:11.607649 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.607630    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cbf3386-2dbf-4f96-8f0a-a7d90fe862df-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g\" (UID: \"5cbf3386-2dbf-4f96-8f0a-a7d90fe862df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g"
Nov 01 06:25:11.669067 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.669029    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvfnq\" (UniqueName: \"kubernetes.io/projected/5cbf3386-2dbf-4f96-8f0a-a7d90fe862df-kube-api-access-nvfnq\") pod \"payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g\" (UID: \"5cbf3386-2dbf-4f96-8f0a-a7d90fe862df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g"
Nov 01 06:25:11.708572 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.708537    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c806c76-debc-4f3d-88dc-da5ec7b978b0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5\" (UID: \"4c806c76-debc-4f3d-88dc-da5ec7b978b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5"
Nov 01 06:25:11.708752 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.708705    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjt8f\" (UniqueName: \"kubernetes.io/projected/4c806c76-debc-4f3d-88dc-da5ec7b978b0-kube-api-access-wjt8f\") pod \"payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5\" (UID: \"4c806c76-debc-4f3d-88dc-da5ec7b978b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5"
Nov 01 06:25:11.709307 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.709286    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c806c76-debc-4f3d-88dc-da5ec7b978b0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5\" (UID: \"4c806c76-debc-4f3d-88dc-da5ec7b978b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5"
Nov 01 06:25:11.770461 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.770425    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjt8f\" (UniqueName: \"kubernetes.io/projected/4c806c76-debc-4f3d-88dc-da5ec7b978b0-kube-api-access-wjt8f\") pod \"payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5\" (UID: \"4c806c76-debc-4f3d-88dc-da5ec7b978b0\") " pod="openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5"
Nov 01 06:25:11.825508 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.825479    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5"
Nov 01 06:25:11.963391 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:11.963357    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g"
Nov 01 06:25:12.209567 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:12.207417    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5"]
Nov 01 06:25:12.338892 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:12.338867    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g"]
Nov 01 06:25:12.339702 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:25:12.339676    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cbf3386_2dbf_4f96_8f0a_a7d90fe862df.slice/crio-44f7f987f7f409d6b1decfad9bd2716e39bcdd390fa26c2331c95daff70462f2 WatchSource:0}: Error finding container 44f7f987f7f409d6b1decfad9bd2716e39bcdd390fa26c2331c95daff70462f2: Status 404 returned error can't find the container with id 44f7f987f7f409d6b1decfad9bd2716e39bcdd390fa26c2331c95daff70462f2
Nov 01 06:25:12.685018 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:12.684980    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g" event={"ID":"5cbf3386-2dbf-4f96-8f0a-a7d90fe862df","Type":"ContainerStarted","Data":"dfdafaabd04108ee8a30e74dc5ddea3d4aa906fcef6c99bc0cbe9218f6bf8f95"}
Nov 01 06:25:12.685202 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:12.685037    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g" event={"ID":"5cbf3386-2dbf-4f96-8f0a-a7d90fe862df","Type":"ContainerStarted","Data":"44f7f987f7f409d6b1decfad9bd2716e39bcdd390fa26c2331c95daff70462f2"}
Nov 01 06:25:12.685202 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:12.685058    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g"
Nov 01 06:25:12.686812 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:12.686787    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5" event={"ID":"4c806c76-debc-4f3d-88dc-da5ec7b978b0","Type":"ContainerStarted","Data":"f8c3e5407926d3505aa74ea5e8446995b119ef69de633de0d81d9be72e21eb18"}
Nov 01 06:25:12.686947 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:12.686833    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5" event={"ID":"4c806c76-debc-4f3d-88dc-da5ec7b978b0","Type":"ContainerStarted","Data":"0933196ab9c39af7576a6280548878c2151160c265932a02413493ddec58ed73"}
Nov 01 06:25:12.686947 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:12.686864    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5"
Nov 01 06:25:12.803344 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:12.803283    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g" podStartSLOduration=1.803269041 podStartE2EDuration="1.803269041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:12.802401729 +0000 UTC m=+2350.013225239" watchObservedRunningTime="2026-11-01 06:25:12.803269041 +0000 UTC m=+2350.014092550"
Nov 01 06:25:12.925781 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:12.925486    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5" podStartSLOduration=1.925466191 podStartE2EDuration="1.925466191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:12.921684569 +0000 UTC m=+2350.132508078" watchObservedRunningTime="2026-11-01 06:25:12.925466191 +0000 UTC m=+2350.136289701"
Nov 01 06:25:13.636049 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:13.636010    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3986bf/agc-cred-6204f08c208e: secret "agc-cred-6204f08c208e" not found
Nov 01 06:25:13.636049 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:13.636041    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4: secret "agc-cred-6204f08c208e" not found
Nov 01 06:25:13.636608 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:13.636104    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-provider-credentials podName:5ec32d57-998e-4646-885b-094f7fbaafd7 nodeName:}" failed. No retries permitted until 2026-11-01 06:25:21.636086495 +0000 UTC m=+2358.846909996 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4" (UID: "5ec32d57-998e-4646-885b-094f7fbaafd7") : secret "agc-cred-6204f08c208e" not found
Nov 01 06:25:14.078129 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:14.078036    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:15.594714 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:15.594679    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:18.866424 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:18.866385    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:21.712403 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:21.712367    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:21.729262 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:21.729233    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3986bf/agc-cred-6204f08c208e: secret "agc-cred-6204f08c208e" not found
Nov 01 06:25:21.729262 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:21.729257    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3986bf/agc-cred-d344d6f1c604: secret "agc-cred-d344d6f1c604" not found
Nov 01 06:25:21.729532 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:21.729270    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4: [secret "agc-cred-6204f08c208e" not found, secret "agc-cred-d344d6f1c604" not found]
Nov 01 06:25:21.729532 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:21.729376    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-provider-credentials podName:5ec32d57-998e-4646-885b-094f7fbaafd7 nodeName:}" failed. No retries permitted until 2026-11-01 06:25:37.729355633 +0000 UTC m=+2374.940179138 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4" (UID: "5ec32d57-998e-4646-885b-094f7fbaafd7") : [secret "agc-cred-6204f08c208e" not found, secret "agc-cred-d344d6f1c604" not found]
Nov 01 06:25:21.856804 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:21.856766    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:22.126206 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:22.126167    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:22.885770 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:22.885738    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"]
Nov 01 06:25:22.890271 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:22.890247    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"
Nov 01 06:25:22.944726 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:22.944695    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc111dd-5efa-428a-80d4-3e12a5064acf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr\" (UID: \"ddc111dd-5efa-428a-80d4-3e12a5064acf\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"
Nov 01 06:25:22.944879 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:22.944815    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc6zp\" (UniqueName: \"kubernetes.io/projected/ddc111dd-5efa-428a-80d4-3e12a5064acf-kube-api-access-gc6zp\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr\" (UID: \"ddc111dd-5efa-428a-80d4-3e12a5064acf\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"
Nov 01 06:25:22.944879 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:22.944844    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ddc111dd-5efa-428a-80d4-3e12a5064acf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr\" (UID: \"ddc111dd-5efa-428a-80d4-3e12a5064acf\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"
Nov 01 06:25:22.944982 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:22.944915    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ddc111dd-5efa-428a-80d4-3e12a5064acf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr\" (UID: \"ddc111dd-5efa-428a-80d4-3e12a5064acf\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"
Nov 01 06:25:22.954860 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:22.954824    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"]
Nov 01 06:25:23.046259 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:23.046222    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc6zp\" (UniqueName: \"kubernetes.io/projected/ddc111dd-5efa-428a-80d4-3e12a5064acf-kube-api-access-gc6zp\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr\" (UID: \"ddc111dd-5efa-428a-80d4-3e12a5064acf\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"
Nov 01 06:25:23.046259 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:23.046262    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ddc111dd-5efa-428a-80d4-3e12a5064acf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr\" (UID: \"ddc111dd-5efa-428a-80d4-3e12a5064acf\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"
Nov 01 06:25:23.046530 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:23.046402    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ddc111dd-5efa-428a-80d4-3e12a5064acf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr\" (UID: \"ddc111dd-5efa-428a-80d4-3e12a5064acf\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"
Nov 01 06:25:23.046530 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:23.046521    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc111dd-5efa-428a-80d4-3e12a5064acf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr\" (UID: \"ddc111dd-5efa-428a-80d4-3e12a5064acf\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"
Nov 01 06:25:23.047096 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:23.047070    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ddc111dd-5efa-428a-80d4-3e12a5064acf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr\" (UID: \"ddc111dd-5efa-428a-80d4-3e12a5064acf\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"
Nov 01 06:25:23.047292 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:23.047275    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc111dd-5efa-428a-80d4-3e12a5064acf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr\" (UID: \"ddc111dd-5efa-428a-80d4-3e12a5064acf\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"
Nov 01 06:25:23.048895 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:23.048871    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ddc111dd-5efa-428a-80d4-3e12a5064acf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr\" (UID: \"ddc111dd-5efa-428a-80d4-3e12a5064acf\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"
Nov 01 06:25:23.108530 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:23.108503    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc6zp\" (UniqueName: \"kubernetes.io/projected/ddc111dd-5efa-428a-80d4-3e12a5064acf-kube-api-access-gc6zp\") pod \"payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr\" (UID: \"ddc111dd-5efa-428a-80d4-3e12a5064acf\") " pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"
Nov 01 06:25:23.205651 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:23.205568    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"
Nov 01 06:25:23.649297 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:23.649266    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"]
Nov 01 06:25:23.649450 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:25:23.649304    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddc111dd_5efa_428a_80d4_3e12a5064acf.slice/crio-a739ad9813a85a31e039b9235e23b713c58e8d089fd4730b0bddf5de7967d112 WatchSource:0}: Error finding container a739ad9813a85a31e039b9235e23b713c58e8d089fd4730b0bddf5de7967d112: Status 404 returned error can't find the container with id a739ad9813a85a31e039b9235e23b713c58e8d089fd4730b0bddf5de7967d112
Nov 01 06:25:23.674990 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:23.674964    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:23.695925 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:23.695906    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g"
Nov 01 06:25:23.697113 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:23.697091    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5"
Nov 01 06:25:23.759786 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:23.759746    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr" event={"ID":"ddc111dd-5efa-428a-80d4-3e12a5064acf","Type":"ContainerStarted","Data":"ea48040fdb4f77eaf31b73a4abe3fee9ef0daea4d295fd2bf336abafca106b16"}
Nov 01 06:25:23.759998 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:23.759816    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr" event={"ID":"ddc111dd-5efa-428a-80d4-3e12a5064acf","Type":"ContainerStarted","Data":"a739ad9813a85a31e039b9235e23b713c58e8d089fd4730b0bddf5de7967d112"}
Nov 01 06:25:23.760122 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:23.760097    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"
Nov 01 06:25:24.053188 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:24.053087    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr" podStartSLOduration=2.053071768 podStartE2EDuration="2.053071768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:24.05140871 +0000 UTC m=+2361.262232220" watchObservedRunningTime="2026-11-01 06:25:24.053071768 +0000 UTC m=+2361.263895277"
Nov 01 06:25:24.875975 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:24.875943    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:25.880221 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:25.880184    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:25:28.434282 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:28.434232    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:29.693686 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:29.693653    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:33.512904 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:33.512857    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:33.719388 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:33.719350    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:33.861519 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:33.861480    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:34.568802 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:34.568759    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:34.780362 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:34.780303    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr" podUID="ddc111dd-5efa-428a-80d4-3e12a5064acf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:25:36.978289 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:36.978243    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:37.118212 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:37.118172    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:37.738029 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:37.737999    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3986bf/agc-cred-6204f08c208e: secret "agc-cred-6204f08c208e" not found
Nov 01 06:25:37.738029 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:37.738027    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3986bf/agc-cred-d344d6f1c604: secret "agc-cred-d344d6f1c604" not found
Nov 01 06:25:37.738029 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:37.738038    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4: [secret "agc-cred-6204f08c208e" not found, secret "agc-cred-d344d6f1c604" not found]
Nov 01 06:25:37.738304 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:37.738122    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-provider-credentials podName:5ec32d57-998e-4646-885b-094f7fbaafd7 nodeName:}" failed. No retries permitted until 2026-11-01 06:26:09.738094059 +0000 UTC m=+2406.948917560 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4" (UID: "5ec32d57-998e-4646-885b-094f7fbaafd7") : [secret "agc-cred-6204f08c208e" not found, secret "agc-cred-d344d6f1c604" not found]
Nov 01 06:25:37.764973 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:37.764933    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"]
Nov 01 06:25:37.769191 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:37.769167    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"
Nov 01 06:25:37.824680 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:37.824635    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"routing-overlay\""
Nov 01 06:25:37.824680 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:37.824657    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"agc-cred-b2178f695dad\""
Nov 01 06:25:37.839300 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:37.839268    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2\" (UID: \"3befe8bb-59cc-47e7-a19f-76f7bea7bce2\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"
Nov 01 06:25:37.839499 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:37.839474    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9vwk\" (UniqueName: \"kubernetes.io/projected/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-kube-api-access-l9vwk\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2\" (UID: \"3befe8bb-59cc-47e7-a19f-76f7bea7bce2\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"
Nov 01 06:25:37.839587 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:37.839573    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2\" (UID: \"3befe8bb-59cc-47e7-a19f-76f7bea7bce2\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"
Nov 01 06:25:37.839658 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:37.839649    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2\" (UID: \"3befe8bb-59cc-47e7-a19f-76f7bea7bce2\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"
Nov 01 06:25:37.848465 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:37.848429    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"agc-cred-bdcae79313a2\""
Nov 01 06:25:37.876149 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:37.876100    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7d25db\""
Nov 01 06:25:37.876391 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:37.876368    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"kube-root-ca.crt\""
Nov 01 06:25:37.876474 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:37.876461    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"payload-processing-external-model-e2e--a835acb2-dockercfg-kqgqq\""
Nov 01 06:25:37.876527 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:37.876516    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7d25db\"/\"openshift-service-ca.crt\""
Nov 01 06:25:37.880719 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:37.880695    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"]
Nov 01 06:25:37.940729 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:37.940688    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2\" (UID: \"3befe8bb-59cc-47e7-a19f-76f7bea7bce2\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"
Nov 01 06:25:37.940900 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:37.940785    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2\" (UID: \"3befe8bb-59cc-47e7-a19f-76f7bea7bce2\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"
Nov 01 06:25:37.940955 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:37.940938    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9vwk\" (UniqueName: \"kubernetes.io/projected/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-kube-api-access-l9vwk\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2\" (UID: \"3befe8bb-59cc-47e7-a19f-76f7bea7bce2\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"
Nov 01 06:25:37.941037 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:37.941021    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2\" (UID: \"3befe8bb-59cc-47e7-a19f-76f7bea7bce2\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"
Nov 01 06:25:37.941576 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:37.941551    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2\" (UID: \"3befe8bb-59cc-47e7-a19f-76f7bea7bce2\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"
Nov 01 06:25:37.941701 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:37.941618    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2\" (UID: \"3befe8bb-59cc-47e7-a19f-76f7bea7bce2\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"
Nov 01 06:25:37.943240 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:37.943222    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2\" (UID: \"3befe8bb-59cc-47e7-a19f-76f7bea7bce2\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"
Nov 01 06:25:38.003275 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:38.003206    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9vwk\" (UniqueName: \"kubernetes.io/projected/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-kube-api-access-l9vwk\") pod \"payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2\" (UID: \"3befe8bb-59cc-47e7-a19f-76f7bea7bce2\") " pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"
Nov 01 06:25:38.083871 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:38.083841    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"
Nov 01 06:25:38.466711 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:38.463084    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"]
Nov 01 06:25:38.490633 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:25:38.490596    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3befe8bb_59cc_47e7_a19f_76f7bea7bce2.slice/crio-a08585da53f2eb9b8819ebddf06bd0f19431530bc4b4e7954c977b040984160e WatchSource:0}: Error finding container a08585da53f2eb9b8819ebddf06bd0f19431530bc4b4e7954c977b040984160e: Status 404 returned error can't find the container with id a08585da53f2eb9b8819ebddf06bd0f19431530bc4b4e7954c977b040984160e
Nov 01 06:25:38.577891 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:38.577849    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:38.871385 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:38.871348    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2" event={"ID":"3befe8bb-59cc-47e7-a19f-76f7bea7bce2","Type":"ContainerStarted","Data":"01c2a06495d6f1d7fc99e4c1c449ac3d7f16b4546aeb03bf97adf31fba1a2ed9"}
Nov 01 06:25:38.871559 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:38.871408    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2" event={"ID":"3befe8bb-59cc-47e7-a19f-76f7bea7bce2","Type":"ContainerStarted","Data":"a08585da53f2eb9b8819ebddf06bd0f19431530bc4b4e7954c977b040984160e"}
Nov 01 06:25:38.871625 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:38.871566    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"
Nov 01 06:25:38.991378 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:38.991304    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2" podStartSLOduration=1.991290638 podStartE2EDuration="1.991290638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:38.989937814 +0000 UTC m=+2376.200761327" watchObservedRunningTime="2026-11-01 06:25:38.991290638 +0000 UTC m=+2376.202114147"
Nov 01 06:25:39.868420 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:39.867063    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:40.018875 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:40.018834    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:40.307184 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:40.307094    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:42.900250 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:42.900218    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"]
Nov 01 06:25:42.901482 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:42.901454    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr" podUID="ddc111dd-5efa-428a-80d4-3e12a5064acf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:25:43.374949 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:43.374913    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:44.320254 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:44.320219    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc"]
Nov 01 06:25:44.430715 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:44.430676    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6"]
Nov 01 06:25:44.683393 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:44.683357    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:44.882286 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:44.882246    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:47.900914 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:47.900875    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr" podUID="ddc111dd-5efa-428a-80d4-3e12a5064acf" containerName="payload-processing" containerID="cri-o://ea48040fdb4f77eaf31b73a4abe3fee9ef0daea4d295fd2bf336abafca106b16" gracePeriod=25
Nov 01 06:25:48.170397 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:48.170264    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"
Nov 01 06:25:48.174544 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:48.174513    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:48.215304 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:48.215272    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddc111dd-5efa-428a-80d4-3e12a5064acf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc111dd-5efa-428a-80d4-3e12a5064acf-plugins-config-volume\") pod \"ddc111dd-5efa-428a-80d4-3e12a5064acf\" (UID: \"ddc111dd-5efa-428a-80d4-3e12a5064acf\") "
Nov 01 06:25:48.215502 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:48.215394    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddc111dd-5efa-428a-80d4-3e12a5064acf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ddc111dd-5efa-428a-80d4-3e12a5064acf-provider-credentials\") pod \"ddc111dd-5efa-428a-80d4-3e12a5064acf\" (UID: \"ddc111dd-5efa-428a-80d4-3e12a5064acf\") "
Nov 01 06:25:48.215573 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:48.215530    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddc111dd-5efa-428a-80d4-3e12a5064acf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ddc111dd-5efa-428a-80d4-3e12a5064acf-routing-overlay-volume\") pod \"ddc111dd-5efa-428a-80d4-3e12a5064acf\" (UID: \"ddc111dd-5efa-428a-80d4-3e12a5064acf\") "
Nov 01 06:25:48.216649 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:48.215765    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddc111dd-5efa-428a-80d4-3e12a5064acf-kube-api-access-gc6zp\" (UniqueName: \"kubernetes.io/projected/ddc111dd-5efa-428a-80d4-3e12a5064acf-kube-api-access-gc6zp\") pod \"ddc111dd-5efa-428a-80d4-3e12a5064acf\" (UID: \"ddc111dd-5efa-428a-80d4-3e12a5064acf\") "
Nov 01 06:25:48.216649 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:48.215785    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc111dd-5efa-428a-80d4-3e12a5064acf-plugins-config-volume" pod "ddc111dd-5efa-428a-80d4-3e12a5064acf" (UID: "ddc111dd-5efa-428a-80d4-3e12a5064acf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:25:48.216649 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:48.216201    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc111dd-5efa-428a-80d4-3e12a5064acf-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:48.216649 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:48.216113    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc111dd-5efa-428a-80d4-3e12a5064acf-routing-overlay-volume" pod "ddc111dd-5efa-428a-80d4-3e12a5064acf" (UID: "ddc111dd-5efa-428a-80d4-3e12a5064acf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:25:48.218296 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:48.218271    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc111dd-5efa-428a-80d4-3e12a5064acf-kube-api-access-gc6zp" pod "ddc111dd-5efa-428a-80d4-3e12a5064acf" (UID: "ddc111dd-5efa-428a-80d4-3e12a5064acf"). InnerVolumeSpecName "kube-api-access-gc6zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:25:48.218484 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:48.218461    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc111dd-5efa-428a-80d4-3e12a5064acf-provider-credentials" pod "ddc111dd-5efa-428a-80d4-3e12a5064acf" (UID: "ddc111dd-5efa-428a-80d4-3e12a5064acf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:25:48.317990 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:48.317952    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc6zp\" (UniqueName: \"kubernetes.io/projected/ddc111dd-5efa-428a-80d4-3e12a5064acf-kube-api-access-gc6zp\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:48.317990 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:48.317994    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ddc111dd-5efa-428a-80d4-3e12a5064acf-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:48.318187 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:48.318011    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ddc111dd-5efa-428a-80d4-3e12a5064acf-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:48.991630 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:48.991593    5619 generic.go:356] "Generic (PLEG): container finished" podID="ddc111dd-5efa-428a-80d4-3e12a5064acf" containerID="ea48040fdb4f77eaf31b73a4abe3fee9ef0daea4d295fd2bf336abafca106b16" exitCode=0
Nov 01 06:25:48.992027 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:48.991634    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"
Nov 01 06:25:48.992027 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:48.991682    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr" event={"ID":"ddc111dd-5efa-428a-80d4-3e12a5064acf","Type":"ContainerDied","Data":"ea48040fdb4f77eaf31b73a4abe3fee9ef0daea4d295fd2bf336abafca106b16"}
Nov 01 06:25:48.992027 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:48.991729    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr" event={"ID":"ddc111dd-5efa-428a-80d4-3e12a5064acf","Type":"ContainerDied","Data":"a739ad9813a85a31e039b9235e23b713c58e8d089fd4730b0bddf5de7967d112"}
Nov 01 06:25:48.992027 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:48.991757    5619 scope.go:122] "RemoveContainer" containerID="ea48040fdb4f77eaf31b73a4abe3fee9ef0daea4d295fd2bf336abafca106b16"
Nov 01 06:25:49.001608 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:49.001587    5619 scope.go:122] "RemoveContainer" containerID="ea48040fdb4f77eaf31b73a4abe3fee9ef0daea4d295fd2bf336abafca106b16"
Nov 01 06:25:49.001904 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:49.001884    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea48040fdb4f77eaf31b73a4abe3fee9ef0daea4d295fd2bf336abafca106b16\": container with ID starting with ea48040fdb4f77eaf31b73a4abe3fee9ef0daea4d295fd2bf336abafca106b16 not found: ID does not exist" containerID="ea48040fdb4f77eaf31b73a4abe3fee9ef0daea4d295fd2bf336abafca106b16"
Nov 01 06:25:49.001950 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:49.001912    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea48040fdb4f77eaf31b73a4abe3fee9ef0daea4d295fd2bf336abafca106b16"} err="failed to get container status \"ea48040fdb4f77eaf31b73a4abe3fee9ef0daea4d295fd2bf336abafca106b16\": rpc error: code = NotFound desc = could not find container \"ea48040fdb4f77eaf31b73a4abe3fee9ef0daea4d295fd2bf336abafca106b16\": container with ID starting with ea48040fdb4f77eaf31b73a4abe3fee9ef0daea4d295fd2bf336abafca106b16 not found: ID does not exist"
Nov 01 06:25:49.159687 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:49.159654    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"]
Nov 01 06:25:49.210191 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:49.210156    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-f6c55x5wr"]
Nov 01 06:25:49.321243 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:49.321107    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc" podUID="a0b83090-6be0-4931-8695-9e313298fefe" containerName="payload-processing" containerID="cri-o://a8ef238bc2f1740b5742d70c235dc85d57cdba5a50bb4d15c67ee34deca78057" gracePeriod=25
Nov 01 06:25:49.431867 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:49.431799    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6" podUID="935d319f-3873-4cb4-b654-bd4dbb8221a3" containerName="payload-pre-processing" containerID="cri-o://85fd7a6f49d03af8f6a0df850bd34eba49decbf15e9174dfe7cb27490eb6449e" gracePeriod=25
Nov 01 06:25:49.578791 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:49.578729    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc"
Nov 01 06:25:49.634677 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:49.634647    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b83090-6be0-4931-8695-9e313298fefe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0b83090-6be0-4931-8695-9e313298fefe-plugins-config-volume\") pod \"a0b83090-6be0-4931-8695-9e313298fefe\" (UID: \"a0b83090-6be0-4931-8695-9e313298fefe\") "
Nov 01 06:25:49.634853 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:49.634743    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b83090-6be0-4931-8695-9e313298fefe-kube-api-access-9tbfb\" (UniqueName: \"kubernetes.io/projected/a0b83090-6be0-4931-8695-9e313298fefe-kube-api-access-9tbfb\") pod \"a0b83090-6be0-4931-8695-9e313298fefe\" (UID: \"a0b83090-6be0-4931-8695-9e313298fefe\") "
Nov 01 06:25:49.635051 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:49.635028    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b83090-6be0-4931-8695-9e313298fefe-plugins-config-volume" pod "a0b83090-6be0-4931-8695-9e313298fefe" (UID: "a0b83090-6be0-4931-8695-9e313298fefe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:25:49.635396 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:49.635374    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0b83090-6be0-4931-8695-9e313298fefe-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:49.636964 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:49.636935    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b83090-6be0-4931-8695-9e313298fefe-kube-api-access-9tbfb" pod "a0b83090-6be0-4931-8695-9e313298fefe" (UID: "a0b83090-6be0-4931-8695-9e313298fefe"). InnerVolumeSpecName "kube-api-access-9tbfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:25:49.701761 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:49.701732    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6"
Nov 01 06:25:49.737193 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:49.737161    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tbfb\" (UniqueName: \"kubernetes.io/projected/a0b83090-6be0-4931-8695-9e313298fefe-kube-api-access-9tbfb\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:49.838287 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:49.838202    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/935d319f-3873-4cb4-b654-bd4dbb8221a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935d319f-3873-4cb4-b654-bd4dbb8221a3-plugins-config-volume\") pod \"935d319f-3873-4cb4-b654-bd4dbb8221a3\" (UID: \"935d319f-3873-4cb4-b654-bd4dbb8221a3\") "
Nov 01 06:25:49.838463 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:49.838357    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/935d319f-3873-4cb4-b654-bd4dbb8221a3-kube-api-access-5ls5z\" (UniqueName: \"kubernetes.io/projected/935d319f-3873-4cb4-b654-bd4dbb8221a3-kube-api-access-5ls5z\") pod \"935d319f-3873-4cb4-b654-bd4dbb8221a3\" (UID: \"935d319f-3873-4cb4-b654-bd4dbb8221a3\") "
Nov 01 06:25:49.838677 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:49.838649    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935d319f-3873-4cb4-b654-bd4dbb8221a3-plugins-config-volume" pod "935d319f-3873-4cb4-b654-bd4dbb8221a3" (UID: "935d319f-3873-4cb4-b654-bd4dbb8221a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:25:49.839099 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:49.839077    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935d319f-3873-4cb4-b654-bd4dbb8221a3-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:49.840471 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:49.840450    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935d319f-3873-4cb4-b654-bd4dbb8221a3-kube-api-access-5ls5z" pod "935d319f-3873-4cb4-b654-bd4dbb8221a3" (UID: "935d319f-3873-4cb4-b654-bd4dbb8221a3"). InnerVolumeSpecName "kube-api-access-5ls5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:25:49.884214 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:49.884184    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2" podUID="3befe8bb-59cc-47e7-a19f-76f7bea7bce2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:25:49.940094 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:49.940066    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ls5z\" (UniqueName: \"kubernetes.io/projected/935d319f-3873-4cb4-b654-bd4dbb8221a3-kube-api-access-5ls5z\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:50.009615 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:50.009576    5619 generic.go:356] "Generic (PLEG): container finished" podID="935d319f-3873-4cb4-b654-bd4dbb8221a3" containerID="85fd7a6f49d03af8f6a0df850bd34eba49decbf15e9174dfe7cb27490eb6449e" exitCode=0
Nov 01 06:25:50.010081 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:50.009637    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6"
Nov 01 06:25:50.010081 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:50.009658    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6" event={"ID":"935d319f-3873-4cb4-b654-bd4dbb8221a3","Type":"ContainerDied","Data":"85fd7a6f49d03af8f6a0df850bd34eba49decbf15e9174dfe7cb27490eb6449e"}
Nov 01 06:25:50.010081 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:50.009733    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6" event={"ID":"935d319f-3873-4cb4-b654-bd4dbb8221a3","Type":"ContainerDied","Data":"6f43c24729ef88c1ff6b99758e66001a6afdae89f7dac3fb59b1cf70144d4fb4"}
Nov 01 06:25:50.010081 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:50.009756    5619 scope.go:122] "RemoveContainer" containerID="85fd7a6f49d03af8f6a0df850bd34eba49decbf15e9174dfe7cb27490eb6449e"
Nov 01 06:25:50.011279 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:50.011260    5619 generic.go:356] "Generic (PLEG): container finished" podID="a0b83090-6be0-4931-8695-9e313298fefe" containerID="a8ef238bc2f1740b5742d70c235dc85d57cdba5a50bb4d15c67ee34deca78057" exitCode=0
Nov 01 06:25:50.011400 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:50.011380    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc" event={"ID":"a0b83090-6be0-4931-8695-9e313298fefe","Type":"ContainerDied","Data":"a8ef238bc2f1740b5742d70c235dc85d57cdba5a50bb4d15c67ee34deca78057"}
Nov 01 06:25:50.011459 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:50.011426    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc" event={"ID":"a0b83090-6be0-4931-8695-9e313298fefe","Type":"ContainerDied","Data":"9258100b6889dda4c7e211335c7d69949f7d4a42525b7702c53f8f19b0bd33b9"}
Nov 01 06:25:50.011518 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:50.011505    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc"
Nov 01 06:25:50.022561 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:50.022546    5619 scope.go:122] "RemoveContainer" containerID="85fd7a6f49d03af8f6a0df850bd34eba49decbf15e9174dfe7cb27490eb6449e"
Nov 01 06:25:50.022827 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:50.022806    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85fd7a6f49d03af8f6a0df850bd34eba49decbf15e9174dfe7cb27490eb6449e\": container with ID starting with 85fd7a6f49d03af8f6a0df850bd34eba49decbf15e9174dfe7cb27490eb6449e not found: ID does not exist" containerID="85fd7a6f49d03af8f6a0df850bd34eba49decbf15e9174dfe7cb27490eb6449e"
Nov 01 06:25:50.022894 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:50.022838    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85fd7a6f49d03af8f6a0df850bd34eba49decbf15e9174dfe7cb27490eb6449e"} err="failed to get container status \"85fd7a6f49d03af8f6a0df850bd34eba49decbf15e9174dfe7cb27490eb6449e\": rpc error: code = NotFound desc = could not find container \"85fd7a6f49d03af8f6a0df850bd34eba49decbf15e9174dfe7cb27490eb6449e\": container with ID starting with 85fd7a6f49d03af8f6a0df850bd34eba49decbf15e9174dfe7cb27490eb6449e not found: ID does not exist"
Nov 01 06:25:50.022894 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:50.022863    5619 scope.go:122] "RemoveContainer" containerID="a8ef238bc2f1740b5742d70c235dc85d57cdba5a50bb4d15c67ee34deca78057"
Nov 01 06:25:50.033052 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:50.033035    5619 scope.go:122] "RemoveContainer" containerID="a8ef238bc2f1740b5742d70c235dc85d57cdba5a50bb4d15c67ee34deca78057"
Nov 01 06:25:50.033349 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:50.033309    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8ef238bc2f1740b5742d70c235dc85d57cdba5a50bb4d15c67ee34deca78057\": container with ID starting with a8ef238bc2f1740b5742d70c235dc85d57cdba5a50bb4d15c67ee34deca78057 not found: ID does not exist" containerID="a8ef238bc2f1740b5742d70c235dc85d57cdba5a50bb4d15c67ee34deca78057"
Nov 01 06:25:50.033420 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:50.033360    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ef238bc2f1740b5742d70c235dc85d57cdba5a50bb4d15c67ee34deca78057"} err="failed to get container status \"a8ef238bc2f1740b5742d70c235dc85d57cdba5a50bb4d15c67ee34deca78057\": rpc error: code = NotFound desc = could not find container \"a8ef238bc2f1740b5742d70c235dc85d57cdba5a50bb4d15c67ee34deca78057\": container with ID starting with a8ef238bc2f1740b5742d70c235dc85d57cdba5a50bb4d15c67ee34deca78057 not found: ID does not exist"
Nov 01 06:25:50.177786 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:50.177750    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6"]
Nov 01 06:25:50.181835 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:50.181811    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-83558e40-5666855b7f-2xdb6"]
Nov 01 06:25:50.292445 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:50.292413    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc"]
Nov 01 06:25:50.296697 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:50.296671    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83558e40-5858d9d48c-wwvvc"]
Nov 01 06:25:50.962196 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:50.962167    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="935d319f-3873-4cb4-b654-bd4dbb8221a3" path="/var/lib/kubelet/pods/935d319f-3873-4cb4-b654-bd4dbb8221a3/volumes"
Nov 01 06:25:50.962580 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:50.962566    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0b83090-6be0-4931-8695-9e313298fefe" path="/var/lib/kubelet/pods/a0b83090-6be0-4931-8695-9e313298fefe/volumes"
Nov 01 06:25:50.962913 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:50.962901    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddc111dd-5efa-428a-80d4-3e12a5064acf" path="/var/lib/kubelet/pods/ddc111dd-5efa-428a-80d4-3e12a5064acf/volumes"
Nov 01 06:25:52.885959 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:52.885926    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:53.096343 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:53.090835    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:53.335561 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:53.335474    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:53.675019 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:53.674984    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:54.444657 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:54.444624    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"]
Nov 01 06:25:54.588620 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:54.588574    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:55.879574 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:25:55.879539    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:25:57.664993 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:57.664957    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:59.445377 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:59.445310    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4" podUID="5ec32d57-998e-4646-885b-094f7fbaafd7" containerName="payload-processing" containerID="cri-o://55b13ca88f8f899b45c35bcf8530483bbf330da8436588c7d2913b160b45f0c0" gracePeriod=25
Nov 01 06:25:59.615175 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:59.615145    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:59.772394 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:59.768749    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:59.815810 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:59.815778    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"
Nov 01 06:25:59.883283 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:59.883227    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2" podUID="3befe8bb-59cc-47e7-a19f-76f7bea7bce2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:25:59.884656 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:59.883719    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-provider-credentials\") pod \"5ec32d57-998e-4646-885b-094f7fbaafd7\" (UID: \"5ec32d57-998e-4646-885b-094f7fbaafd7\") "
Nov 01 06:25:59.884656 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:59.883789    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ec32d57-998e-4646-885b-094f7fbaafd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec32d57-998e-4646-885b-094f7fbaafd7-plugins-config-volume\") pod \"5ec32d57-998e-4646-885b-094f7fbaafd7\" (UID: \"5ec32d57-998e-4646-885b-094f7fbaafd7\") "
Nov 01 06:25:59.884656 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:59.883962    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-kube-api-access-zjpzb\" (UniqueName: \"kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-kube-api-access-zjpzb\") pod \"5ec32d57-998e-4646-885b-094f7fbaafd7\" (UID: \"5ec32d57-998e-4646-885b-094f7fbaafd7\") "
Nov 01 06:25:59.884656 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:59.884003    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ec32d57-998e-4646-885b-094f7fbaafd7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5ec32d57-998e-4646-885b-094f7fbaafd7-routing-overlay-volume\") pod \"5ec32d57-998e-4646-885b-094f7fbaafd7\" (UID: \"5ec32d57-998e-4646-885b-094f7fbaafd7\") "
Nov 01 06:25:59.884656 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:59.884509    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec32d57-998e-4646-885b-094f7fbaafd7-plugins-config-volume" pod "5ec32d57-998e-4646-885b-094f7fbaafd7" (UID: "5ec32d57-998e-4646-885b-094f7fbaafd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:25:59.885162 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:59.885132    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec32d57-998e-4646-885b-094f7fbaafd7-routing-overlay-volume" pod "5ec32d57-998e-4646-885b-094f7fbaafd7" (UID: "5ec32d57-998e-4646-885b-094f7fbaafd7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:25:59.886613 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:59.886576    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-provider-credentials" pod "5ec32d57-998e-4646-885b-094f7fbaafd7" (UID: "5ec32d57-998e-4646-885b-094f7fbaafd7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:25:59.886805 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:59.886781    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-kube-api-access-zjpzb" pod "5ec32d57-998e-4646-885b-094f7fbaafd7" (UID: "5ec32d57-998e-4646-885b-094f7fbaafd7"). InnerVolumeSpecName "kube-api-access-zjpzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:25:59.920204 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:59.920167    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:25:59.985390 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:59.985355    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:59.985390 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:59.985383    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec32d57-998e-4646-885b-094f7fbaafd7-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:59.985390 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:59.985397    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjpzb\" (UniqueName: \"kubernetes.io/projected/5ec32d57-998e-4646-885b-094f7fbaafd7-kube-api-access-zjpzb\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:25:59.985685 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:25:59.985406    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5ec32d57-998e-4646-885b-094f7fbaafd7-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:00.085793 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:00.085697    5619 generic.go:356] "Generic (PLEG): container finished" podID="5ec32d57-998e-4646-885b-094f7fbaafd7" containerID="55b13ca88f8f899b45c35bcf8530483bbf330da8436588c7d2913b160b45f0c0" exitCode=0
Nov 01 06:26:00.085793 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:00.085759    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"
Nov 01 06:26:00.085980 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:00.085795    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4" event={"ID":"5ec32d57-998e-4646-885b-094f7fbaafd7","Type":"ContainerDied","Data":"55b13ca88f8f899b45c35bcf8530483bbf330da8436588c7d2913b160b45f0c0"}
Nov 01 06:26:00.085980 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:00.085876    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4" event={"ID":"5ec32d57-998e-4646-885b-094f7fbaafd7","Type":"ContainerDied","Data":"ba09403cb714ad8d3cae26085b6555fc045b0fbf6d01e837c3d7de1697e144fe"}
Nov 01 06:26:00.085980 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:00.085909    5619 scope.go:122] "RemoveContainer" containerID="55b13ca88f8f899b45c35bcf8530483bbf330da8436588c7d2913b160b45f0c0"
Nov 01 06:26:00.096834 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:00.096814    5619 scope.go:122] "RemoveContainer" containerID="55b13ca88f8f899b45c35bcf8530483bbf330da8436588c7d2913b160b45f0c0"
Nov 01 06:26:00.097505 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:26:00.097482    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b13ca88f8f899b45c35bcf8530483bbf330da8436588c7d2913b160b45f0c0\": container with ID starting with 55b13ca88f8f899b45c35bcf8530483bbf330da8436588c7d2913b160b45f0c0 not found: ID does not exist" containerID="55b13ca88f8f899b45c35bcf8530483bbf330da8436588c7d2913b160b45f0c0"
Nov 01 06:26:00.097646 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:00.097622    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b13ca88f8f899b45c35bcf8530483bbf330da8436588c7d2913b160b45f0c0"} err="failed to get container status \"55b13ca88f8f899b45c35bcf8530483bbf330da8436588c7d2913b160b45f0c0\": rpc error: code = NotFound desc = could not find container \"55b13ca88f8f899b45c35bcf8530483bbf330da8436588c7d2913b160b45f0c0\": container with ID starting with 55b13ca88f8f899b45c35bcf8530483bbf330da8436588c7d2913b160b45f0c0 not found: ID does not exist"
Nov 01 06:26:00.255438 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:00.255396    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"]
Nov 01 06:26:00.261198 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:00.261162    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3986bf/payload-processing-external-model-e2e-extmod-b-3986bf-644ftw5n4"]
Nov 01 06:26:00.962962 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:00.962930    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ec32d57-998e-4646-885b-094f7fbaafd7" path="/var/lib/kubelet/pods/5ec32d57-998e-4646-885b-094f7fbaafd7/volumes"
Nov 01 06:26:02.128298 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.128262    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"]
Nov 01 06:26:02.128862 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.128846    5619 cpu_manager.go:414] "containerMap: removing container" podUID="5ec32d57-998e-4646-885b-094f7fbaafd7" containerName="payload-processing"
Nov 01 06:26:02.128862 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.128861    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec32d57-998e-4646-885b-094f7fbaafd7" containerName="payload-processing"
Nov 01 06:26:02.128994 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.128885    5619 cpu_manager.go:414] "containerMap: removing container" podUID="ddc111dd-5efa-428a-80d4-3e12a5064acf" containerName="payload-processing"
Nov 01 06:26:02.128994 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.128891    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddc111dd-5efa-428a-80d4-3e12a5064acf" containerName="payload-processing"
Nov 01 06:26:02.128994 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.128901    5619 cpu_manager.go:414] "containerMap: removing container" podUID="935d319f-3873-4cb4-b654-bd4dbb8221a3" containerName="payload-pre-processing"
Nov 01 06:26:02.128994 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.128906    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="935d319f-3873-4cb4-b654-bd4dbb8221a3" containerName="payload-pre-processing"
Nov 01 06:26:02.128994 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.128939    5619 cpu_manager.go:414] "containerMap: removing container" podUID="a0b83090-6be0-4931-8695-9e313298fefe" containerName="payload-processing"
Nov 01 06:26:02.128994 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.128945    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b83090-6be0-4931-8695-9e313298fefe" containerName="payload-processing"
Nov 01 06:26:02.129312 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.129019    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddc111dd-5efa-428a-80d4-3e12a5064acf" containerName="payload-processing"
Nov 01 06:26:02.129312 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.129029    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="935d319f-3873-4cb4-b654-bd4dbb8221a3" containerName="payload-pre-processing"
Nov 01 06:26:02.129312 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.129036    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b83090-6be0-4931-8695-9e313298fefe" containerName="payload-processing"
Nov 01 06:26:02.129312 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.129044    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec32d57-998e-4646-885b-094f7fbaafd7" containerName="payload-processing"
Nov 01 06:26:02.133502 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.133482    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"
Nov 01 06:26:02.187042 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.187007    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f13905-serving-cert\""
Nov 01 06:26:02.212699 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.212652    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1312ccd3-4c59-4cd1-b087-93c9aabcd152-metrics-certs\") pod \"maas-api-e2e-api-a-f13905-859955f999-824pt\" (UID: \"1312ccd3-4c59-4cd1-b087-93c9aabcd152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"
Nov 01 06:26:02.212892 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.212738    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1312ccd3-4c59-4cd1-b087-93c9aabcd152-maas-api-tls\") pod \"maas-api-e2e-api-a-f13905-859955f999-824pt\" (UID: \"1312ccd3-4c59-4cd1-b087-93c9aabcd152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"
Nov 01 06:26:02.212892 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.212872    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggzsc\" (UniqueName: \"kubernetes.io/projected/1312ccd3-4c59-4cd1-b087-93c9aabcd152-kube-api-access-ggzsc\") pod \"maas-api-e2e-api-a-f13905-859955f999-824pt\" (UID: \"1312ccd3-4c59-4cd1-b087-93c9aabcd152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"
Nov 01 06:26:02.213049 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.213032    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1312ccd3-4c59-4cd1-b087-93c9aabcd152-openshift-service-ca\") pod \"maas-api-e2e-api-a-f13905-859955f999-824pt\" (UID: \"1312ccd3-4c59-4cd1-b087-93c9aabcd152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"
Nov 01 06:26:02.240475 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.240441    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"]
Nov 01 06:26:02.314582 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.314546    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggzsc\" (UniqueName: \"kubernetes.io/projected/1312ccd3-4c59-4cd1-b087-93c9aabcd152-kube-api-access-ggzsc\") pod \"maas-api-e2e-api-a-f13905-859955f999-824pt\" (UID: \"1312ccd3-4c59-4cd1-b087-93c9aabcd152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"
Nov 01 06:26:02.314791 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.314633    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1312ccd3-4c59-4cd1-b087-93c9aabcd152-openshift-service-ca\") pod \"maas-api-e2e-api-a-f13905-859955f999-824pt\" (UID: \"1312ccd3-4c59-4cd1-b087-93c9aabcd152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"
Nov 01 06:26:02.314791 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.314707    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1312ccd3-4c59-4cd1-b087-93c9aabcd152-metrics-certs\") pod \"maas-api-e2e-api-a-f13905-859955f999-824pt\" (UID: \"1312ccd3-4c59-4cd1-b087-93c9aabcd152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"
Nov 01 06:26:02.314791 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.314747    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1312ccd3-4c59-4cd1-b087-93c9aabcd152-maas-api-tls\") pod \"maas-api-e2e-api-a-f13905-859955f999-824pt\" (UID: \"1312ccd3-4c59-4cd1-b087-93c9aabcd152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"
Nov 01 06:26:02.314978 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:26:02.314861    5619 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-serving-cert: secret "maas-api-e2e-api-a-f13905-serving-cert" not found
Nov 01 06:26:02.314978 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:26:02.314928    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1312ccd3-4c59-4cd1-b087-93c9aabcd152-maas-api-tls podName:1312ccd3-4c59-4cd1-b087-93c9aabcd152 nodeName:}" failed. No retries permitted until 2026-11-01 06:26:02.814906568 +0000 UTC m=+2400.025730071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1312ccd3-4c59-4cd1-b087-93c9aabcd152-maas-api-tls") pod "maas-api-e2e-api-a-f13905-859955f999-824pt" (UID: "1312ccd3-4c59-4cd1-b087-93c9aabcd152") : secret "maas-api-e2e-api-a-f13905-serving-cert" not found
Nov 01 06:26:02.315298 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.315278    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1312ccd3-4c59-4cd1-b087-93c9aabcd152-openshift-service-ca\") pod \"maas-api-e2e-api-a-f13905-859955f999-824pt\" (UID: \"1312ccd3-4c59-4cd1-b087-93c9aabcd152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"
Nov 01 06:26:02.317134 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.317111    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1312ccd3-4c59-4cd1-b087-93c9aabcd152-metrics-certs\") pod \"maas-api-e2e-api-a-f13905-859955f999-824pt\" (UID: \"1312ccd3-4c59-4cd1-b087-93c9aabcd152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"
Nov 01 06:26:02.326419 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.326390    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:02.400739 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.400652    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggzsc\" (UniqueName: \"kubernetes.io/projected/1312ccd3-4c59-4cd1-b087-93c9aabcd152-kube-api-access-ggzsc\") pod \"maas-api-e2e-api-a-f13905-859955f999-824pt\" (UID: \"1312ccd3-4c59-4cd1-b087-93c9aabcd152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"
Nov 01 06:26:02.523712 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.523664    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:02.671599 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.671510    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:02.820317 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.820282    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1312ccd3-4c59-4cd1-b087-93c9aabcd152-maas-api-tls\") pod \"maas-api-e2e-api-a-f13905-859955f999-824pt\" (UID: \"1312ccd3-4c59-4cd1-b087-93c9aabcd152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"
Nov 01 06:26:02.822899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:02.822869    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1312ccd3-4c59-4cd1-b087-93c9aabcd152-maas-api-tls\") pod \"maas-api-e2e-api-a-f13905-859955f999-824pt\" (UID: \"1312ccd3-4c59-4cd1-b087-93c9aabcd152\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"
Nov 01 06:26:03.048569 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:03.048491    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"
Nov 01 06:26:03.438250 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:03.438218    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"]
Nov 01 06:26:03.439143 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:26:03.439106    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1312ccd3_4c59_4cd1_b087_93c9aabcd152.slice/crio-7f5f94139f281e6ebb010a87a76a60a32d992e4edfa83a6a10a23eec895f325f WatchSource:0}: Error finding container 7f5f94139f281e6ebb010a87a76a60a32d992e4edfa83a6a10a23eec895f325f: Status 404 returned error can't find the container with id 7f5f94139f281e6ebb010a87a76a60a32d992e4edfa83a6a10a23eec895f325f
Nov 01 06:26:04.117185 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:04.117152    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt" event={"ID":"1312ccd3-4c59-4cd1-b087-93c9aabcd152","Type":"ContainerStarted","Data":"7f5f94139f281e6ebb010a87a76a60a32d992e4edfa83a6a10a23eec895f325f"}
Nov 01 06:26:04.297767 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:04.297729    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:04.495352 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:04.495301    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:05.135484 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:05.135449    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt" event={"ID":"1312ccd3-4c59-4cd1-b087-93c9aabcd152","Type":"ContainerStarted","Data":"3fd48e9d111882b79078c9558af5a0a2b54fa5423e73f8561d01cce02754e8c3"}
Nov 01 06:26:05.135804 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:05.135610    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"
Nov 01 06:26:05.255573 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:05.255523    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt" podStartSLOduration=2.325768575 podStartE2EDuration="3.255510008s" totalImagesPullingTime="929.741433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:26:05.254367413 +0000 UTC m=+2402.465190987" watchObservedRunningTime="2026-11-01 06:26:05.255510008 +0000 UTC m=+2402.466333533"
Nov 01 06:26:05.671906 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:05.671872    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:05.819415 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:05.819366    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:06.790421 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:06.790385    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:06.996714 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:06.996667    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:07.270604 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:07.270565    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:09.883570 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:09.883531    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2" podUID="3befe8bb-59cc-47e7-a19f-76f7bea7bce2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:26:10.156848 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:10.156761    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:11.148532 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.148505    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"
Nov 01 06:26:11.322227 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.322194    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"]
Nov 01 06:26:11.328491 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.328463    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"
Nov 01 06:26:11.382062 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.381805    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-76ab178ef073\""
Nov 01 06:26:11.382303 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.382276    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 06:26:11.382506 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.382483    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 06:26:11.382794 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.382774    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 06:26:11.382920 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.382864    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 06:26:11.383042 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.383023    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-k4wk5\""
Nov 01 06:26:11.385047 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.385025    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"]
Nov 01 06:26:11.447088 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.447011    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-routing-overlay-volume\") pod \"payload-processing-external-model-59f5fcbf6-tdc59\" (UID: \"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa\") " pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"
Nov 01 06:26:11.447088 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.447059    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-provider-credentials\") pod \"payload-processing-external-model-59f5fcbf6-tdc59\" (UID: \"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa\") " pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"
Nov 01 06:26:11.447260 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.447187    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpcbk\" (UniqueName: \"kubernetes.io/projected/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-kube-api-access-dpcbk\") pod \"payload-processing-external-model-59f5fcbf6-tdc59\" (UID: \"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa\") " pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"
Nov 01 06:26:11.447260 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.447229    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-plugins-config-volume\") pod \"payload-processing-external-model-59f5fcbf6-tdc59\" (UID: \"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa\") " pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"
Nov 01 06:26:11.548100 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.548063    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-routing-overlay-volume\") pod \"payload-processing-external-model-59f5fcbf6-tdc59\" (UID: \"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa\") " pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"
Nov 01 06:26:11.548280 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.548107    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-provider-credentials\") pod \"payload-processing-external-model-59f5fcbf6-tdc59\" (UID: \"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa\") " pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"
Nov 01 06:26:11.548280 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.548203    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpcbk\" (UniqueName: \"kubernetes.io/projected/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-kube-api-access-dpcbk\") pod \"payload-processing-external-model-59f5fcbf6-tdc59\" (UID: \"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa\") " pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"
Nov 01 06:26:11.548427 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.548381    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-plugins-config-volume\") pod \"payload-processing-external-model-59f5fcbf6-tdc59\" (UID: \"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa\") " pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"
Nov 01 06:26:11.548758 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.548738    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-routing-overlay-volume\") pod \"payload-processing-external-model-59f5fcbf6-tdc59\" (UID: \"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa\") " pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"
Nov 01 06:26:11.548997 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.548981    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-plugins-config-volume\") pod \"payload-processing-external-model-59f5fcbf6-tdc59\" (UID: \"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa\") " pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"
Nov 01 06:26:11.550598 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.550568    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-provider-credentials\") pod \"payload-processing-external-model-59f5fcbf6-tdc59\" (UID: \"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa\") " pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"
Nov 01 06:26:11.610135 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.610103    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpcbk\" (UniqueName: \"kubernetes.io/projected/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-kube-api-access-dpcbk\") pod \"payload-processing-external-model-59f5fcbf6-tdc59\" (UID: \"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa\") " pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"
Nov 01 06:26:11.646680 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:11.646649    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"
Nov 01 06:26:12.025400 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:12.025366    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"]
Nov 01 06:26:12.026747 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:26:12.026723    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1900c0_9e4b_43b1_b2a9_9b017f3cd1aa.slice/crio-46ef663300bacb0157869c90ee8215f424d918ae6d22c289512b6115f4ec1090 WatchSource:0}: Error finding container 46ef663300bacb0157869c90ee8215f424d918ae6d22c289512b6115f4ec1090: Status 404 returned error can't find the container with id 46ef663300bacb0157869c90ee8215f424d918ae6d22c289512b6115f4ec1090
Nov 01 06:26:12.069232 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:12.069190    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:12.182711 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:12.182676    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59" event={"ID":"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa","Type":"ContainerStarted","Data":"8fc31e2467493ae56205df9735c9333fb0b328cbd6a1048ae210e1c97fd31cbd"}
Nov 01 06:26:12.183134 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:12.182725    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59" event={"ID":"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa","Type":"ContainerStarted","Data":"46ef663300bacb0157869c90ee8215f424d918ae6d22c289512b6115f4ec1090"}
Nov 01 06:26:12.183134 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:12.182872    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"
Nov 01 06:26:12.288401 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:12.287465    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:12.312935 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:12.310081    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59" podStartSLOduration=1.310060231 podStartE2EDuration="1.310060231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:26:12.301100097 +0000 UTC m=+2409.511923607" watchObservedRunningTime="2026-11-01 06:26:12.310060231 +0000 UTC m=+2409.520883741"
Nov 01 06:26:15.154108 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:15.154064    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"]
Nov 01 06:26:15.155424 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:15.155397    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2" podUID="3befe8bb-59cc-47e7-a19f-76f7bea7bce2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:26:16.920122 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:16.920084    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:20.155392 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:20.155347    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2" podUID="3befe8bb-59cc-47e7-a19f-76f7bea7bce2" containerName="payload-processing" containerID="cri-o://01c2a06495d6f1d7fc99e4c1c449ac3d7f16b4546aeb03bf97adf31fba1a2ed9" gracePeriod=25
Nov 01 06:26:20.429692 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:20.429667    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"
Nov 01 06:26:20.476893 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:20.476863    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-provider-credentials\") pod \"3befe8bb-59cc-47e7-a19f-76f7bea7bce2\" (UID: \"3befe8bb-59cc-47e7-a19f-76f7bea7bce2\") "
Nov 01 06:26:20.477039 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:20.476918    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-kube-api-access-l9vwk\" (UniqueName: \"kubernetes.io/projected/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-kube-api-access-l9vwk\") pod \"3befe8bb-59cc-47e7-a19f-76f7bea7bce2\" (UID: \"3befe8bb-59cc-47e7-a19f-76f7bea7bce2\") "
Nov 01 06:26:20.477039 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:20.476942    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-plugins-config-volume\") pod \"3befe8bb-59cc-47e7-a19f-76f7bea7bce2\" (UID: \"3befe8bb-59cc-47e7-a19f-76f7bea7bce2\") "
Nov 01 06:26:20.477168 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:20.477150    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-routing-overlay-volume\") pod \"3befe8bb-59cc-47e7-a19f-76f7bea7bce2\" (UID: \"3befe8bb-59cc-47e7-a19f-76f7bea7bce2\") "
Nov 01 06:26:20.477504 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:20.477472    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-plugins-config-volume" pod "3befe8bb-59cc-47e7-a19f-76f7bea7bce2" (UID: "3befe8bb-59cc-47e7-a19f-76f7bea7bce2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:20.477608 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:20.477511    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-routing-overlay-volume" pod "3befe8bb-59cc-47e7-a19f-76f7bea7bce2" (UID: "3befe8bb-59cc-47e7-a19f-76f7bea7bce2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:20.477822 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:20.477798    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:20.477940 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:20.477828    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:20.479239 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:20.479214    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-provider-credentials" pod "3befe8bb-59cc-47e7-a19f-76f7bea7bce2" (UID: "3befe8bb-59cc-47e7-a19f-76f7bea7bce2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:20.479343 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:20.479249    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-kube-api-access-l9vwk" pod "3befe8bb-59cc-47e7-a19f-76f7bea7bce2" (UID: "3befe8bb-59cc-47e7-a19f-76f7bea7bce2"). InnerVolumeSpecName "kube-api-access-l9vwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:20.578842 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:20.578812    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9vwk\" (UniqueName: \"kubernetes.io/projected/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-kube-api-access-l9vwk\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:20.578842 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:20.578839    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3befe8bb-59cc-47e7-a19f-76f7bea7bce2-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:21.242296 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:21.242222    5619 generic.go:356] "Generic (PLEG): container finished" podID="3befe8bb-59cc-47e7-a19f-76f7bea7bce2" containerID="01c2a06495d6f1d7fc99e4c1c449ac3d7f16b4546aeb03bf97adf31fba1a2ed9" exitCode=0
Nov 01 06:26:21.242296 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:21.242259    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2" event={"ID":"3befe8bb-59cc-47e7-a19f-76f7bea7bce2","Type":"ContainerDied","Data":"01c2a06495d6f1d7fc99e4c1c449ac3d7f16b4546aeb03bf97adf31fba1a2ed9"}
Nov 01 06:26:21.242296 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:21.242266    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"
Nov 01 06:26:21.242790 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:21.242309    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2" event={"ID":"3befe8bb-59cc-47e7-a19f-76f7bea7bce2","Type":"ContainerDied","Data":"a08585da53f2eb9b8819ebddf06bd0f19431530bc4b4e7954c977b040984160e"}
Nov 01 06:26:21.242790 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:21.242354    5619 scope.go:122] "RemoveContainer" containerID="01c2a06495d6f1d7fc99e4c1c449ac3d7f16b4546aeb03bf97adf31fba1a2ed9"
Nov 01 06:26:21.252542 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:21.252518    5619 scope.go:122] "RemoveContainer" containerID="01c2a06495d6f1d7fc99e4c1c449ac3d7f16b4546aeb03bf97adf31fba1a2ed9"
Nov 01 06:26:21.252841 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:26:21.252821    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c2a06495d6f1d7fc99e4c1c449ac3d7f16b4546aeb03bf97adf31fba1a2ed9\": container with ID starting with 01c2a06495d6f1d7fc99e4c1c449ac3d7f16b4546aeb03bf97adf31fba1a2ed9 not found: ID does not exist" containerID="01c2a06495d6f1d7fc99e4c1c449ac3d7f16b4546aeb03bf97adf31fba1a2ed9"
Nov 01 06:26:21.252916 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:21.252851    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c2a06495d6f1d7fc99e4c1c449ac3d7f16b4546aeb03bf97adf31fba1a2ed9"} err="failed to get container status \"01c2a06495d6f1d7fc99e4c1c449ac3d7f16b4546aeb03bf97adf31fba1a2ed9\": rpc error: code = NotFound desc = could not find container \"01c2a06495d6f1d7fc99e4c1c449ac3d7f16b4546aeb03bf97adf31fba1a2ed9\": container with ID starting with 01c2a06495d6f1d7fc99e4c1c449ac3d7f16b4546aeb03bf97adf31fba1a2ed9 not found: ID does not exist"
Nov 01 06:26:21.410045 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:21.410015    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"]
Nov 01 06:26:21.462509 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:21.462472    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7d25db/payload-processing-external-model-e2e-extmod-b-7d25db-7876q48n2"]
Nov 01 06:26:21.673851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:21.673813    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:22.963994 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:22.963963    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3befe8bb-59cc-47e7-a19f-76f7bea7bce2" path="/var/lib/kubelet/pods/3befe8bb-59cc-47e7-a19f-76f7bea7bce2/volumes"
Nov 01 06:26:23.192146 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:23.192114    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"
Nov 01 06:26:25.326243 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:25.326200    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:25.903446 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:25.903408    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:26.047638 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:26:26.047571    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:26:27.652670 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:27.652634    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:28.037002 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:28.036913    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:28.184912 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:28.184854    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:29.647555 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:29.647519    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"]
Nov 01 06:26:29.647965 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:29.647775    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7" podUID="4c0ea9c5-f467-4031-8bd3-8a724d3848e9" containerName="istio-proxy" containerID="cri-o://6c32baee4921ca84080a606fc8d81ebed89fa61c372406828b2c1018fec3da20" gracePeriod=30
Nov 01 06:26:29.667519 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:29.667480    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:30.056389 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:30.056268    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:30.236938 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:30.236899    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:30.825361 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:30.825305    5619 patch_prober.go:29] interesting pod/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:26:30.825746 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:30.825384    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7" podUID="4c0ea9c5-f467-4031-8bd3-8a724d3848e9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:26:31.275869 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:31.275834    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:31.460808 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:31.460772    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:33.089692 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:33.089658    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:34.512553 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:34.512521    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:34.674063 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:34.674020    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:34.825871 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:34.825831    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:34.933430 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:34.933405    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:26:35.033983 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.033910    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-envoy\") pod \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") "
Nov 01 06:26:35.033983 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.033957    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-kube-api-access-xf2f8\" (UniqueName: \"kubernetes.io/projected/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-kube-api-access-xf2f8\") pod \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") "
Nov 01 06:26:35.033983 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.033977    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-data\") pod \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") "
Nov 01 06:26:35.034357 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.034014    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-podinfo\") pod \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") "
Nov 01 06:26:35.034357 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.034046    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-credential-socket\") pod \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") "
Nov 01 06:26:35.034357 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.034066    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istiod-ca-cert\") pod \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") "
Nov 01 06:26:35.034357 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.034149    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-workload-certs\") pod \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") "
Nov 01 06:26:35.034357 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.034178    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-token\" (UniqueName: \"kubernetes.io/projected/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-token\") pod \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") "
Nov 01 06:26:35.034357 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.034213    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-workload-socket\") pod \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\" (UID: \"4c0ea9c5-f467-4031-8bd3-8a724d3848e9\") "
Nov 01 06:26:35.034712 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.034370    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-credential-socket" pod "4c0ea9c5-f467-4031-8bd3-8a724d3848e9" (UID: "4c0ea9c5-f467-4031-8bd3-8a724d3848e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:26:35.034712 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.034505    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-workload-certs" pod "4c0ea9c5-f467-4031-8bd3-8a724d3848e9" (UID: "4c0ea9c5-f467-4031-8bd3-8a724d3848e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:26:35.034712 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.034551    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istiod-ca-cert" pod "4c0ea9c5-f467-4031-8bd3-8a724d3848e9" (UID: "4c0ea9c5-f467-4031-8bd3-8a724d3848e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:35.034884 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.034710    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-data" pod "4c0ea9c5-f467-4031-8bd3-8a724d3848e9" (UID: "4c0ea9c5-f467-4031-8bd3-8a724d3848e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:26:35.034884 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.034780    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-workload-socket" pod "4c0ea9c5-f467-4031-8bd3-8a724d3848e9" (UID: "4c0ea9c5-f467-4031-8bd3-8a724d3848e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:26:35.035015 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.034994    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:35.035079 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.035023    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:35.035079 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.035039    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:35.035079 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.035054    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:35.035079 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.035068    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:35.036309 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.036285    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-envoy" pod "4c0ea9c5-f467-4031-8bd3-8a724d3848e9" (UID: "4c0ea9c5-f467-4031-8bd3-8a724d3848e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:26:35.036474 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.036380    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-podinfo" pod "4c0ea9c5-f467-4031-8bd3-8a724d3848e9" (UID: "4c0ea9c5-f467-4031-8bd3-8a724d3848e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:26:35.036587 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.036569    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-kube-api-access-xf2f8" pod "4c0ea9c5-f467-4031-8bd3-8a724d3848e9" (UID: "4c0ea9c5-f467-4031-8bd3-8a724d3848e9"). InnerVolumeSpecName "kube-api-access-xf2f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:35.036665 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.036631    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-token" pod "4c0ea9c5-f467-4031-8bd3-8a724d3848e9" (UID: "4c0ea9c5-f467-4031-8bd3-8a724d3848e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:35.136712 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.136673    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:35.136712 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.136703    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:35.136712 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.136716    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:35.136943 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.136729    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf2f8\" (UniqueName: \"kubernetes.io/projected/4c0ea9c5-f467-4031-8bd3-8a724d3848e9-kube-api-access-xf2f8\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:35.359211 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.359177    5619 generic.go:356] "Generic (PLEG): container finished" podID="4c0ea9c5-f467-4031-8bd3-8a724d3848e9" containerID="6c32baee4921ca84080a606fc8d81ebed89fa61c372406828b2c1018fec3da20" exitCode=0
Nov 01 06:26:35.359406 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.359230    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"
Nov 01 06:26:35.359406 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.359252    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7" event={"ID":"4c0ea9c5-f467-4031-8bd3-8a724d3848e9","Type":"ContainerDied","Data":"6c32baee4921ca84080a606fc8d81ebed89fa61c372406828b2c1018fec3da20"}
Nov 01 06:26:35.359406 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.359313    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7" event={"ID":"4c0ea9c5-f467-4031-8bd3-8a724d3848e9","Type":"ContainerDied","Data":"6c149dc793fbddd1a79934414379b82999dc49d0bfd75db21501a4809a84aae7"}
Nov 01 06:26:35.359406 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.359353    5619 scope.go:122] "RemoveContainer" containerID="6c32baee4921ca84080a606fc8d81ebed89fa61c372406828b2c1018fec3da20"
Nov 01 06:26:35.369656 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.369636    5619 scope.go:122] "RemoveContainer" containerID="ea2804a86cf5b3abc43a00fbd348ac97197ae12b0ca5c47b7034912cd9775904"
Nov 01 06:26:35.379151 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.379129    5619 scope.go:122] "RemoveContainer" containerID="6c32baee4921ca84080a606fc8d81ebed89fa61c372406828b2c1018fec3da20"
Nov 01 06:26:35.379458 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:26:35.379435    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c32baee4921ca84080a606fc8d81ebed89fa61c372406828b2c1018fec3da20\": container with ID starting with 6c32baee4921ca84080a606fc8d81ebed89fa61c372406828b2c1018fec3da20 not found: ID does not exist" containerID="6c32baee4921ca84080a606fc8d81ebed89fa61c372406828b2c1018fec3da20"
Nov 01 06:26:35.379572 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.379466    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c32baee4921ca84080a606fc8d81ebed89fa61c372406828b2c1018fec3da20"} err="failed to get container status \"6c32baee4921ca84080a606fc8d81ebed89fa61c372406828b2c1018fec3da20\": rpc error: code = NotFound desc = could not find container \"6c32baee4921ca84080a606fc8d81ebed89fa61c372406828b2c1018fec3da20\": container with ID starting with 6c32baee4921ca84080a606fc8d81ebed89fa61c372406828b2c1018fec3da20 not found: ID does not exist"
Nov 01 06:26:35.379572 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.379486    5619 scope.go:122] "RemoveContainer" containerID="ea2804a86cf5b3abc43a00fbd348ac97197ae12b0ca5c47b7034912cd9775904"
Nov 01 06:26:35.379737 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:26:35.379717    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2804a86cf5b3abc43a00fbd348ac97197ae12b0ca5c47b7034912cd9775904\": container with ID starting with ea2804a86cf5b3abc43a00fbd348ac97197ae12b0ca5c47b7034912cd9775904 not found: ID does not exist" containerID="ea2804a86cf5b3abc43a00fbd348ac97197ae12b0ca5c47b7034912cd9775904"
Nov 01 06:26:35.379794 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.379742    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2804a86cf5b3abc43a00fbd348ac97197ae12b0ca5c47b7034912cd9775904"} err="failed to get container status \"ea2804a86cf5b3abc43a00fbd348ac97197ae12b0ca5c47b7034912cd9775904\": rpc error: code = NotFound desc = could not find container \"ea2804a86cf5b3abc43a00fbd348ac97197ae12b0ca5c47b7034912cd9775904\": container with ID starting with ea2804a86cf5b3abc43a00fbd348ac97197ae12b0ca5c47b7034912cd9775904 not found: ID does not exist"
Nov 01 06:26:35.536574 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.536543    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"]
Nov 01 06:26:35.587653 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:35.587623    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-83558e40-openshift-default-77b9cd585-qpfx7"]
Nov 01 06:26:36.963445 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:36.963416    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c0ea9c5-f467-4031-8bd3-8a724d3848e9" path="/var/lib/kubelet/pods/4c0ea9c5-f467-4031-8bd3-8a724d3848e9/volumes"
Nov 01 06:26:37.729180 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:37.729141    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:37.882686 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:37.882649    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:38.039470 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:38.039364    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:38.462022 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:38.461986    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:38.692245 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:38.692206    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:39.141314 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:39.141276    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:41.853205 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:41.853126    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:43.512375 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:43.512283    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:44.603349 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:44.603293    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:44.766891 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:44.766844    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:45.124843 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:45.124799    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:45.293445 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:45.293410    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:45.484604 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:45.484512    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:46.373584 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:46.373548    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:48.382289 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:48.382243    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:52.328635 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:52.328596    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:52.393270 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:52.393235    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"]
Nov 01 06:26:52.393525 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:52.393501    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28" podUID="d452a9ec-fe57-41b2-be6f-26ed3a942f32" containerName="maas-api" containerID="cri-o://8400c396af40d27bca36da9217ae032cf47ea47f46e95a02e1d8f306de487342" gracePeriod=30
Nov 01 06:26:52.523858 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:52.523826    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:52.527083 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:52.527053    5619 generic.go:356] "Generic (PLEG): container finished" podID="d452a9ec-fe57-41b2-be6f-26ed3a942f32" containerID="8400c396af40d27bca36da9217ae032cf47ea47f46e95a02e1d8f306de487342" exitCode=0
Nov 01 06:26:52.527205 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:52.527122    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28" event={"ID":"d452a9ec-fe57-41b2-be6f-26ed3a942f32","Type":"ContainerDied","Data":"8400c396af40d27bca36da9217ae032cf47ea47f46e95a02e1d8f306de487342"}
Nov 01 06:26:52.666395 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:52.666362    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"
Nov 01 06:26:52.767660 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:52.767626    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d452a9ec-fe57-41b2-be6f-26ed3a942f32-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d452a9ec-fe57-41b2-be6f-26ed3a942f32-maas-api-tls\") pod \"d452a9ec-fe57-41b2-be6f-26ed3a942f32\" (UID: \"d452a9ec-fe57-41b2-be6f-26ed3a942f32\") "
Nov 01 06:26:52.767852 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:52.767762    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d452a9ec-fe57-41b2-be6f-26ed3a942f32-kube-api-access-vmd9p\" (UniqueName: \"kubernetes.io/projected/d452a9ec-fe57-41b2-be6f-26ed3a942f32-kube-api-access-vmd9p\") pod \"d452a9ec-fe57-41b2-be6f-26ed3a942f32\" (UID: \"d452a9ec-fe57-41b2-be6f-26ed3a942f32\") "
Nov 01 06:26:52.767852 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:52.767826    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d452a9ec-fe57-41b2-be6f-26ed3a942f32-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d452a9ec-fe57-41b2-be6f-26ed3a942f32-metrics-certs\") pod \"d452a9ec-fe57-41b2-be6f-26ed3a942f32\" (UID: \"d452a9ec-fe57-41b2-be6f-26ed3a942f32\") "
Nov 01 06:26:52.767974 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:52.767931    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d452a9ec-fe57-41b2-be6f-26ed3a942f32-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d452a9ec-fe57-41b2-be6f-26ed3a942f32-openshift-service-ca\") pod \"d452a9ec-fe57-41b2-be6f-26ed3a942f32\" (UID: \"d452a9ec-fe57-41b2-be6f-26ed3a942f32\") "
Nov 01 06:26:52.768716 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:52.768687    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d452a9ec-fe57-41b2-be6f-26ed3a942f32-openshift-service-ca" pod "d452a9ec-fe57-41b2-be6f-26ed3a942f32" (UID: "d452a9ec-fe57-41b2-be6f-26ed3a942f32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:52.771962 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:52.771928    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d452a9ec-fe57-41b2-be6f-26ed3a942f32-metrics-certs" pod "d452a9ec-fe57-41b2-be6f-26ed3a942f32" (UID: "d452a9ec-fe57-41b2-be6f-26ed3a942f32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:26:52.775776 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:52.775741    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d452a9ec-fe57-41b2-be6f-26ed3a942f32-kube-api-access-vmd9p" pod "d452a9ec-fe57-41b2-be6f-26ed3a942f32" (UID: "d452a9ec-fe57-41b2-be6f-26ed3a942f32"). InnerVolumeSpecName "kube-api-access-vmd9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:52.785442 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:52.784126    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d452a9ec-fe57-41b2-be6f-26ed3a942f32-maas-api-tls" pod "d452a9ec-fe57-41b2-be6f-26ed3a942f32" (UID: "d452a9ec-fe57-41b2-be6f-26ed3a942f32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:26:52.869547 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:52.869514    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmd9p\" (UniqueName: \"kubernetes.io/projected/d452a9ec-fe57-41b2-be6f-26ed3a942f32-kube-api-access-vmd9p\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:52.869547 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:52.869544    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d452a9ec-fe57-41b2-be6f-26ed3a942f32-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:52.869547 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:52.869554    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d452a9ec-fe57-41b2-be6f-26ed3a942f32-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:52.869780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:52.869563    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d452a9ec-fe57-41b2-be6f-26ed3a942f32-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:52.933389 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:52.933301    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz"]
Nov 01 06:26:52.987766 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:52.987736    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266"]
Nov 01 06:26:53.088991 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:53.088958    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:53.235133 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:53.235057    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:53.550137 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:53.550033    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28" event={"ID":"d452a9ec-fe57-41b2-be6f-26ed3a942f32","Type":"ContainerDied","Data":"4b0607c6866ecaf71addbd6b6037150852b56c763bf3104744b8047a7d18c16d"}
Nov 01 06:26:53.550137 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:53.550059    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"
Nov 01 06:26:53.550684 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:53.550120    5619 scope.go:122] "RemoveContainer" containerID="8400c396af40d27bca36da9217ae032cf47ea47f46e95a02e1d8f306de487342"
Nov 01 06:26:53.717615 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:53.717581    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"]
Nov 01 06:26:53.721585 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:53.721557    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7e551a7-d8d9f988d-c7p28"]
Nov 01 06:26:54.516674 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:54.516633    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:54.963545 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:54.963514    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d452a9ec-fe57-41b2-be6f-26ed3a942f32" path="/var/lib/kubelet/pods/d452a9ec-fe57-41b2-be6f-26ed3a942f32/volumes"
Nov 01 06:26:55.880746 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:26:55.880714    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:26:56.334373 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:56.334270    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:56.528414 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:56.528382    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:57.918472 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:57.918436    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:57.934214 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:57.934182    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz" podUID="4cb4a057-3d46-441d-bb00-8057e63c9d37" containerName="payload-processing" containerID="cri-o://1d8529fba4d4882c9a5e4c98b63c478a6ce2e837e46839983551c674309a4d2c" gracePeriod=25
Nov 01 06:26:57.988927 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:57.988872    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266" podUID="451b691f-2ec5-48a8-b54a-11da6c61fd64" containerName="payload-pre-processing" containerID="cri-o://119b8722f05052db20af98038c026a18d8ba05b4aa1ebe182f3734ae64b98c6c" gracePeriod=25
Nov 01 06:26:58.064516 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.064482    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:58.271182 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.271156    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz"
Nov 01 06:26:58.306535 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.306512    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266"
Nov 01 06:26:58.434885 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.434856    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/451b691f-2ec5-48a8-b54a-11da6c61fd64-kube-api-access-v2pbr\" (UniqueName: \"kubernetes.io/projected/451b691f-2ec5-48a8-b54a-11da6c61fd64-kube-api-access-v2pbr\") pod \"451b691f-2ec5-48a8-b54a-11da6c61fd64\" (UID: \"451b691f-2ec5-48a8-b54a-11da6c61fd64\") "
Nov 01 06:26:58.435124 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.434929    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cb4a057-3d46-441d-bb00-8057e63c9d37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb4a057-3d46-441d-bb00-8057e63c9d37-plugins-config-volume\") pod \"4cb4a057-3d46-441d-bb00-8057e63c9d37\" (UID: \"4cb4a057-3d46-441d-bb00-8057e63c9d37\") "
Nov 01 06:26:58.435124 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.434984    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb4a057-3d46-441d-bb00-8057e63c9d37-kube-api-access-89tf7\" (UniqueName: \"kubernetes.io/projected/4cb4a057-3d46-441d-bb00-8057e63c9d37-kube-api-access-89tf7\") pod \"4cb4a057-3d46-441d-bb00-8057e63c9d37\" (UID: \"4cb4a057-3d46-441d-bb00-8057e63c9d37\") "
Nov 01 06:26:58.435124 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.435043    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/451b691f-2ec5-48a8-b54a-11da6c61fd64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/451b691f-2ec5-48a8-b54a-11da6c61fd64-plugins-config-volume\") pod \"451b691f-2ec5-48a8-b54a-11da6c61fd64\" (UID: \"451b691f-2ec5-48a8-b54a-11da6c61fd64\") "
Nov 01 06:26:58.435591 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.435447    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb4a057-3d46-441d-bb00-8057e63c9d37-plugins-config-volume" pod "4cb4a057-3d46-441d-bb00-8057e63c9d37" (UID: "4cb4a057-3d46-441d-bb00-8057e63c9d37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:58.435776 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.435640    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/451b691f-2ec5-48a8-b54a-11da6c61fd64-plugins-config-volume" pod "451b691f-2ec5-48a8-b54a-11da6c61fd64" (UID: "451b691f-2ec5-48a8-b54a-11da6c61fd64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:26:58.437156 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.437131    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/451b691f-2ec5-48a8-b54a-11da6c61fd64-kube-api-access-v2pbr" pod "451b691f-2ec5-48a8-b54a-11da6c61fd64" (UID: "451b691f-2ec5-48a8-b54a-11da6c61fd64"). InnerVolumeSpecName "kube-api-access-v2pbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:58.437262 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.437172    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb4a057-3d46-441d-bb00-8057e63c9d37-kube-api-access-89tf7" pod "4cb4a057-3d46-441d-bb00-8057e63c9d37" (UID: "4cb4a057-3d46-441d-bb00-8057e63c9d37"). InnerVolumeSpecName "kube-api-access-89tf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:26:58.536662 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.536581    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2pbr\" (UniqueName: \"kubernetes.io/projected/451b691f-2ec5-48a8-b54a-11da6c61fd64-kube-api-access-v2pbr\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:58.536662 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.536620    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb4a057-3d46-441d-bb00-8057e63c9d37-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:58.536662 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.536636    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89tf7\" (UniqueName: \"kubernetes.io/projected/4cb4a057-3d46-441d-bb00-8057e63c9d37-kube-api-access-89tf7\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:58.536662 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.536650    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/451b691f-2ec5-48a8-b54a-11da6c61fd64-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:26:58.588896 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.588865    5619 generic.go:356] "Generic (PLEG): container finished" podID="451b691f-2ec5-48a8-b54a-11da6c61fd64" containerID="119b8722f05052db20af98038c026a18d8ba05b4aa1ebe182f3734ae64b98c6c" exitCode=0
Nov 01 06:26:58.589060 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.588917    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266"
Nov 01 06:26:58.589060 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.588950    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266" event={"ID":"451b691f-2ec5-48a8-b54a-11da6c61fd64","Type":"ContainerDied","Data":"119b8722f05052db20af98038c026a18d8ba05b4aa1ebe182f3734ae64b98c6c"}
Nov 01 06:26:58.589060 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.589009    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266" event={"ID":"451b691f-2ec5-48a8-b54a-11da6c61fd64","Type":"ContainerDied","Data":"a279ba0253704794ed776e807c6a68d088e95b9f59cffdec15ce41f3ea085f4d"}
Nov 01 06:26:58.589060 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.589041    5619 scope.go:122] "RemoveContainer" containerID="119b8722f05052db20af98038c026a18d8ba05b4aa1ebe182f3734ae64b98c6c"
Nov 01 06:26:58.590487 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.590470    5619 generic.go:356] "Generic (PLEG): container finished" podID="4cb4a057-3d46-441d-bb00-8057e63c9d37" containerID="1d8529fba4d4882c9a5e4c98b63c478a6ce2e837e46839983551c674309a4d2c" exitCode=0
Nov 01 06:26:58.590585 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.590514    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz" event={"ID":"4cb4a057-3d46-441d-bb00-8057e63c9d37","Type":"ContainerDied","Data":"1d8529fba4d4882c9a5e4c98b63c478a6ce2e837e46839983551c674309a4d2c"}
Nov 01 06:26:58.590585 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.590552    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz" event={"ID":"4cb4a057-3d46-441d-bb00-8057e63c9d37","Type":"ContainerDied","Data":"4549b00ba98fe9a5b3c1484e05ebacd89cbd50fa3339d4d40517f8092ea15382"}
Nov 01 06:26:58.590710 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.590616    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz"
Nov 01 06:26:58.599983 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.599952    5619 scope.go:122] "RemoveContainer" containerID="119b8722f05052db20af98038c026a18d8ba05b4aa1ebe182f3734ae64b98c6c"
Nov 01 06:26:58.600276 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:26:58.600245    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"119b8722f05052db20af98038c026a18d8ba05b4aa1ebe182f3734ae64b98c6c\": container with ID starting with 119b8722f05052db20af98038c026a18d8ba05b4aa1ebe182f3734ae64b98c6c not found: ID does not exist" containerID="119b8722f05052db20af98038c026a18d8ba05b4aa1ebe182f3734ae64b98c6c"
Nov 01 06:26:58.600406 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.600283    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"119b8722f05052db20af98038c026a18d8ba05b4aa1ebe182f3734ae64b98c6c"} err="failed to get container status \"119b8722f05052db20af98038c026a18d8ba05b4aa1ebe182f3734ae64b98c6c\": rpc error: code = NotFound desc = could not find container \"119b8722f05052db20af98038c026a18d8ba05b4aa1ebe182f3734ae64b98c6c\": container with ID starting with 119b8722f05052db20af98038c026a18d8ba05b4aa1ebe182f3734ae64b98c6c not found: ID does not exist"
Nov 01 06:26:58.600406 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.600310    5619 scope.go:122] "RemoveContainer" containerID="1d8529fba4d4882c9a5e4c98b63c478a6ce2e837e46839983551c674309a4d2c"
Nov 01 06:26:58.609161 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.609147    5619 scope.go:122] "RemoveContainer" containerID="1d8529fba4d4882c9a5e4c98b63c478a6ce2e837e46839983551c674309a4d2c"
Nov 01 06:26:58.609447 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:26:58.609429    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8529fba4d4882c9a5e4c98b63c478a6ce2e837e46839983551c674309a4d2c\": container with ID starting with 1d8529fba4d4882c9a5e4c98b63c478a6ce2e837e46839983551c674309a4d2c not found: ID does not exist" containerID="1d8529fba4d4882c9a5e4c98b63c478a6ce2e837e46839983551c674309a4d2c"
Nov 01 06:26:58.609506 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.609454    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8529fba4d4882c9a5e4c98b63c478a6ce2e837e46839983551c674309a4d2c"} err="failed to get container status \"1d8529fba4d4882c9a5e4c98b63c478a6ce2e837e46839983551c674309a4d2c\": rpc error: code = NotFound desc = could not find container \"1d8529fba4d4882c9a5e4c98b63c478a6ce2e837e46839983551c674309a4d2c\": container with ID starting with 1d8529fba4d4882c9a5e4c98b63c478a6ce2e837e46839983551c674309a4d2c not found: ID does not exist"
Nov 01 06:26:58.757905 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.757871    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz"]
Nov 01 06:26:58.762970 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.762939    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c7e551a7-c979b94f5-55grz"]
Nov 01 06:26:58.873816 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.872726    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266"]
Nov 01 06:26:58.876656 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.876633    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c7e551a7-6ffc5878d7-5c266"]
Nov 01 06:26:58.964062 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.964027    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="451b691f-2ec5-48a8-b54a-11da6c61fd64" path="/var/lib/kubelet/pods/451b691f-2ec5-48a8-b54a-11da6c61fd64/volumes"
Nov 01 06:26:58.964468 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:58.964453    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cb4a057-3d46-441d-bb00-8057e63c9d37" path="/var/lib/kubelet/pods/4cb4a057-3d46-441d-bb00-8057e63c9d37/volumes"
Nov 01 06:26:59.406051 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:59.406009    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:59.600597 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:59.600568    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:26:59.956900 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:26:59.956860    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:00.422203 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:00.422167    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:00.576190 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:00.576150    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:00.722245 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:00.722142    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:00.877864 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:00.877822    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:01.843813 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.843782    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"]
Nov 01 06:27:01.844402 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.844306    5619 cpu_manager.go:414] "containerMap: removing container" podUID="451b691f-2ec5-48a8-b54a-11da6c61fd64" containerName="payload-pre-processing"
Nov 01 06:27:01.844402 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.844337    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="451b691f-2ec5-48a8-b54a-11da6c61fd64" containerName="payload-pre-processing"
Nov 01 06:27:01.844402 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.844365    5619 cpu_manager.go:414] "containerMap: removing container" podUID="3befe8bb-59cc-47e7-a19f-76f7bea7bce2" containerName="payload-processing"
Nov 01 06:27:01.844402 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.844370    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3befe8bb-59cc-47e7-a19f-76f7bea7bce2" containerName="payload-processing"
Nov 01 06:27:01.844402 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.844378    5619 cpu_manager.go:414] "containerMap: removing container" podUID="d452a9ec-fe57-41b2-be6f-26ed3a942f32" containerName="maas-api"
Nov 01 06:27:01.844402 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.844383    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d452a9ec-fe57-41b2-be6f-26ed3a942f32" containerName="maas-api"
Nov 01 06:27:01.844402 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.844397    5619 cpu_manager.go:414] "containerMap: removing container" podUID="4c0ea9c5-f467-4031-8bd3-8a724d3848e9" containerName="istio-proxy"
Nov 01 06:27:01.844402 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.844403    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c0ea9c5-f467-4031-8bd3-8a724d3848e9" containerName="istio-proxy"
Nov 01 06:27:01.844684 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.844413    5619 cpu_manager.go:414] "containerMap: removing container" podUID="4c0ea9c5-f467-4031-8bd3-8a724d3848e9" containerName="istio-proxy"
Nov 01 06:27:01.844684 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.844418    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c0ea9c5-f467-4031-8bd3-8a724d3848e9" containerName="istio-proxy"
Nov 01 06:27:01.844684 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.844427    5619 cpu_manager.go:414] "containerMap: removing container" podUID="4cb4a057-3d46-441d-bb00-8057e63c9d37" containerName="payload-processing"
Nov 01 06:27:01.844684 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.844432    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb4a057-3d46-441d-bb00-8057e63c9d37" containerName="payload-processing"
Nov 01 06:27:01.844684 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.844510    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="d452a9ec-fe57-41b2-be6f-26ed3a942f32" containerName="maas-api"
Nov 01 06:27:01.844684 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.844521    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="451b691f-2ec5-48a8-b54a-11da6c61fd64" containerName="payload-pre-processing"
Nov 01 06:27:01.844684 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.844528    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="3befe8bb-59cc-47e7-a19f-76f7bea7bce2" containerName="payload-processing"
Nov 01 06:27:01.844684 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.844535    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c0ea9c5-f467-4031-8bd3-8a724d3848e9" containerName="istio-proxy"
Nov 01 06:27:01.844684 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.844542    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c0ea9c5-f467-4031-8bd3-8a724d3848e9" containerName="istio-proxy"
Nov 01 06:27:01.844684 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.844549    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cb4a057-3d46-441d-bb00-8057e63c9d37" containerName="payload-processing"
Nov 01 06:27:01.869347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.849310    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"
Nov 01 06:27:01.907571 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.907536    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj5hc\" (UniqueName: \"kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-kube-api-access-sj5hc\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7\" (UID: \"8c54c622-970a-4d9a-8bb6-06803bcf1a42\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"
Nov 01 06:27:01.907848 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.907820    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7\" (UID: \"8c54c622-970a-4d9a-8bb6-06803bcf1a42\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"
Nov 01 06:27:01.907995 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.907978    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c54c622-970a-4d9a-8bb6-06803bcf1a42-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7\" (UID: \"8c54c622-970a-4d9a-8bb6-06803bcf1a42\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"
Nov 01 06:27:01.908204 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.908185    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c54c622-970a-4d9a-8bb6-06803bcf1a42-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7\" (UID: \"8c54c622-970a-4d9a-8bb6-06803bcf1a42\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"
Nov 01 06:27:01.929935 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.929897    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"]
Nov 01 06:27:01.977348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.960738    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8f45a2\""
Nov 01 06:27:01.977348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.960991    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"payload-processing-external-model-e2e--cb1d2723-dockercfg-zfdw5\""
Nov 01 06:27:01.977348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.961071    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"agc-cred-47cacd68a67a\""
Nov 01 06:27:01.977348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.961751    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"routing-overlay\""
Nov 01 06:27:01.977348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.961863    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"kube-root-ca.crt\""
Nov 01 06:27:01.977348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:01.961958    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"openshift-service-ca.crt\""
Nov 01 06:27:02.009222 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:02.009177    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7\" (UID: \"8c54c622-970a-4d9a-8bb6-06803bcf1a42\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"
Nov 01 06:27:02.010415 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:02.010387    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c54c622-970a-4d9a-8bb6-06803bcf1a42-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7\" (UID: \"8c54c622-970a-4d9a-8bb6-06803bcf1a42\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"
Nov 01 06:27:02.021343 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:02.018575    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c54c622-970a-4d9a-8bb6-06803bcf1a42-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7\" (UID: \"8c54c622-970a-4d9a-8bb6-06803bcf1a42\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"
Nov 01 06:27:02.021343 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:02.018818    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj5hc\" (UniqueName: \"kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-kube-api-access-sj5hc\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7\" (UID: \"8c54c622-970a-4d9a-8bb6-06803bcf1a42\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"
Nov 01 06:27:02.021343 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:02.011288    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c54c622-970a-4d9a-8bb6-06803bcf1a42-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7\" (UID: \"8c54c622-970a-4d9a-8bb6-06803bcf1a42\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"
Nov 01 06:27:02.021343 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:02.020967    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c54c622-970a-4d9a-8bb6-06803bcf1a42-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7\" (UID: \"8c54c622-970a-4d9a-8bb6-06803bcf1a42\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"
Nov 01 06:27:02.031673 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:02.031574    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7\" (UID: \"8c54c622-970a-4d9a-8bb6-06803bcf1a42\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"
Nov 01 06:27:02.094180 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:02.094090    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj5hc\" (UniqueName: \"kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-kube-api-access-sj5hc\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7\" (UID: \"8c54c622-970a-4d9a-8bb6-06803bcf1a42\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"
Nov 01 06:27:02.219347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:02.218616    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"
Nov 01 06:27:02.729527 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:02.728676    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"]
Nov 01 06:27:02.734904 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:27:02.734872    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c54c622_970a_4d9a_8bb6_06803bcf1a42.slice/crio-2d778be3a307f43fbf8df6ab269251242c48ebb639268d4700a245410ccca295 WatchSource:0}: Error finding container 2d778be3a307f43fbf8df6ab269251242c48ebb639268d4700a245410ccca295: Status 404 returned error can't find the container with id 2d778be3a307f43fbf8df6ab269251242c48ebb639268d4700a245410ccca295
Nov 01 06:27:03.345977 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:03.345942    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-x9mzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:27:03.346889 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:03.346520    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj" podUID="d299e7a4-892c-404f-8346-91bcae60326e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:27:03.739566 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:03.739477    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7" event={"ID":"8c54c622-970a-4d9a-8bb6-06803bcf1a42","Type":"ContainerStarted","Data":"0480c7e59751150338fa79c2f31348e37203981f76e1566501bdda9e4d870b2f"}
Nov 01 06:27:03.739566 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:03.739554    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7" event={"ID":"8c54c622-970a-4d9a-8bb6-06803bcf1a42","Type":"ContainerStarted","Data":"2d778be3a307f43fbf8df6ab269251242c48ebb639268d4700a245410ccca295"}
Nov 01 06:27:03.739745 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:03.739579    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"
Nov 01 06:27:03.862340 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:03.862278    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7" podStartSLOduration=2.862264837 podStartE2EDuration="2.862264837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:27:03.859383793 +0000 UTC m=+2461.070207303" watchObservedRunningTime="2026-11-01 06:27:03.862264837 +0000 UTC m=+2461.073088346"
Nov 01 06:27:05.176728 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:05.176691    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:07.106183 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:07.106148    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:07.259949 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:07.259913    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:07.561486 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:07.561404    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:07.761396 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:07.761359    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:08.929006 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:08.928961    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:09.229711 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:09.229619    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:09.374725 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:09.374681    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:10.317446 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:10.317411    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:10.467110 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:10.467067    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"]
Nov 01 06:27:10.473478 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:10.473450    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"
Nov 01 06:27:10.514672 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:10.514632    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:10.525882 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:10.525852    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07ef354-09ac-41d1-af4d-fd904e5fef4f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx\" (UID: \"b07ef354-09ac-41d1-af4d-fd904e5fef4f\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"
Nov 01 06:27:10.526137 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:10.526098    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57wct\" (UniqueName: \"kubernetes.io/projected/b07ef354-09ac-41d1-af4d-fd904e5fef4f-kube-api-access-57wct\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx\" (UID: \"b07ef354-09ac-41d1-af4d-fd904e5fef4f\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"
Nov 01 06:27:10.526232 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:10.526138    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b07ef354-09ac-41d1-af4d-fd904e5fef4f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx\" (UID: \"b07ef354-09ac-41d1-af4d-fd904e5fef4f\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"
Nov 01 06:27:10.526279 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:10.526251    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b07ef354-09ac-41d1-af4d-fd904e5fef4f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx\" (UID: \"b07ef354-09ac-41d1-af4d-fd904e5fef4f\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"
Nov 01 06:27:10.529413 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:10.529391    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"]
Nov 01 06:27:10.552688 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:10.552654    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"e2e-cfl-fd3b7b-provider-cred\""
Nov 01 06:27:10.627342 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:10.627278    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b07ef354-09ac-41d1-af4d-fd904e5fef4f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx\" (UID: \"b07ef354-09ac-41d1-af4d-fd904e5fef4f\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"
Nov 01 06:27:10.627547 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:10.627514    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07ef354-09ac-41d1-af4d-fd904e5fef4f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx\" (UID: \"b07ef354-09ac-41d1-af4d-fd904e5fef4f\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"
Nov 01 06:27:10.627624 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:10.627574    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57wct\" (UniqueName: \"kubernetes.io/projected/b07ef354-09ac-41d1-af4d-fd904e5fef4f-kube-api-access-57wct\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx\" (UID: \"b07ef354-09ac-41d1-af4d-fd904e5fef4f\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"
Nov 01 06:27:10.627624 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:10.627610    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b07ef354-09ac-41d1-af4d-fd904e5fef4f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx\" (UID: \"b07ef354-09ac-41d1-af4d-fd904e5fef4f\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"
Nov 01 06:27:10.631229 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:10.629250    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b07ef354-09ac-41d1-af4d-fd904e5fef4f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx\" (UID: \"b07ef354-09ac-41d1-af4d-fd904e5fef4f\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"
Nov 01 06:27:10.632227 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:10.632206    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07ef354-09ac-41d1-af4d-fd904e5fef4f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx\" (UID: \"b07ef354-09ac-41d1-af4d-fd904e5fef4f\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"
Nov 01 06:27:10.633782 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:10.633755    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b07ef354-09ac-41d1-af4d-fd904e5fef4f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx\" (UID: \"b07ef354-09ac-41d1-af4d-fd904e5fef4f\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"
Nov 01 06:27:10.694618 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:10.694582    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57wct\" (UniqueName: \"kubernetes.io/projected/b07ef354-09ac-41d1-af4d-fd904e5fef4f-kube-api-access-57wct\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx\" (UID: \"b07ef354-09ac-41d1-af4d-fd904e5fef4f\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"
Nov 01 06:27:10.788240 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:10.788210    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"
Nov 01 06:27:11.170260 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:11.170231    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"]
Nov 01 06:27:11.171246 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:27:11.171215    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07ef354_09ac_41d1_af4d_fd904e5fef4f.slice/crio-0ee7180bbd5e32618a5dcd34208b2472edd5a02bcc65339f004fbccef1322a62 WatchSource:0}: Error finding container 0ee7180bbd5e32618a5dcd34208b2472edd5a02bcc65339f004fbccef1322a62: Status 404 returned error can't find the container with id 0ee7180bbd5e32618a5dcd34208b2472edd5a02bcc65339f004fbccef1322a62
Nov 01 06:27:11.790915 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:11.790878    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" event={"ID":"b07ef354-09ac-41d1-af4d-fd904e5fef4f","Type":"ContainerStarted","Data":"2e9c8e12e0b838630105ee7e956255f27ffc93c601e4bb72b66b60c963db38bf"}
Nov 01 06:27:11.791460 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:11.790934    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" event={"ID":"b07ef354-09ac-41d1-af4d-fd904e5fef4f","Type":"ContainerStarted","Data":"0ee7180bbd5e32618a5dcd34208b2472edd5a02bcc65339f004fbccef1322a62"}
Nov 01 06:27:11.791460 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:11.790958    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"
Nov 01 06:27:11.910051 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:11.909991    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podStartSLOduration=1.9099755539999999 podStartE2EDuration="1.909975554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:27:11.908580934 +0000 UTC m=+2469.119404445" watchObservedRunningTime="2026-11-01 06:27:11.909975554 +0000 UTC m=+2469.120799063"
Nov 01 06:27:13.937909 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:13.937874    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:14.085393 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:14.085351    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:14.749585 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:14.749551    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"
Nov 01 06:27:14.881251 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:14.881221    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-47cacd68a67a: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:27:14.881500 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:14.881472    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:27:14.881651 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:14.881555    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials podName:8c54c622-970a-4d9a-8bb6-06803bcf1a42 nodeName:}" failed. No retries permitted until 2026-11-01 06:27:15.381535248 +0000 UTC m=+2472.592358748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7" (UID: "8c54c622-970a-4d9a-8bb6-06803bcf1a42") : secret "agc-cred-47cacd68a67a" not found
Nov 01 06:27:15.178657 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:15.178622    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:15.388686 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:15.388656    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-47cacd68a67a: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:27:15.388686 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:15.388681    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:27:15.388954 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:15.388757    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials podName:8c54c622-970a-4d9a-8bb6-06803bcf1a42 nodeName:}" failed. No retries permitted until 2026-11-01 06:27:16.388737626 +0000 UTC m=+2473.599561112 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7" (UID: "8c54c622-970a-4d9a-8bb6-06803bcf1a42") : secret "agc-cred-47cacd68a67a" not found
Nov 01 06:27:16.399828 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:16.399796    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-47cacd68a67a: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:27:16.399828 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:16.399821    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:27:16.400407 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:16.399873    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials podName:8c54c622-970a-4d9a-8bb6-06803bcf1a42 nodeName:}" failed. No retries permitted until 2026-11-01 06:27:18.399858438 +0000 UTC m=+2475.610681924 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7" (UID: "8c54c622-970a-4d9a-8bb6-06803bcf1a42") : secret "agc-cred-47cacd68a67a" not found
Nov 01 06:27:16.574418 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:16.574383    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:18.424141 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:18.424106    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-47cacd68a67a: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:27:18.424141 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:18.424139    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:27:18.424612 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:18.424209    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials podName:8c54c622-970a-4d9a-8bb6-06803bcf1a42 nodeName:}" failed. No retries permitted until 2026-11-01 06:27:22.424188894 +0000 UTC m=+2479.635012396 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7" (UID: "8c54c622-970a-4d9a-8bb6-06803bcf1a42") : secret "agc-cred-47cacd68a67a" not found
Nov 01 06:27:19.883809 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:19.883764    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:22.469886 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:22.469854    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-47cacd68a67a: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:27:22.469886 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:22.469883    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:27:22.470354 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:22.469946    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials podName:8c54c622-970a-4d9a-8bb6-06803bcf1a42 nodeName:}" failed. No retries permitted until 2026-11-01 06:27:30.469930446 +0000 UTC m=+2487.680753932 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7" (UID: "8c54c622-970a-4d9a-8bb6-06803bcf1a42") : secret "agc-cred-47cacd68a67a" not found
Nov 01 06:27:22.802439 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:22.802358    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:27:24.590488 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:24.590451    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:25.686368 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:25.686307    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:25.880432 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:25.880398    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:27:27.310843 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:27.310800    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:27.864357 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:27.863916    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:28.216230 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:28.216137    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:29.019825 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:29.019787    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"]
Nov 01 06:27:29.034542 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:29.034508    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"
Nov 01 06:27:29.047603 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:29.047574    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxwbp\" (UniqueName: \"kubernetes.io/projected/10a53273-5b35-4b72-a946-7253763e3168-kube-api-access-zxwbp\") pod \"maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f\" (UID: \"10a53273-5b35-4b72-a946-7253763e3168\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"
Nov 01 06:27:29.047790 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:29.047765    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a53273-5b35-4b72-a946-7253763e3168-metrics-certs\") pod \"maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f\" (UID: \"10a53273-5b35-4b72-a946-7253763e3168\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"
Nov 01 06:27:29.047940 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:29.047924    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10a53273-5b35-4b72-a946-7253763e3168-maas-api-tls\") pod \"maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f\" (UID: \"10a53273-5b35-4b72-a946-7253763e3168\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"
Nov 01 06:27:29.047986 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:29.047962    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a53273-5b35-4b72-a946-7253763e3168-openshift-service-ca\") pod \"maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f\" (UID: \"10a53273-5b35-4b72-a946-7253763e3168\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"
Nov 01 06:27:29.082097 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:29.082068    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"]
Nov 01 06:27:29.088916 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:29.088442    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ce346503-serving-cert\""
Nov 01 06:27:29.149292 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:29.149258    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10a53273-5b35-4b72-a946-7253763e3168-maas-api-tls\") pod \"maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f\" (UID: \"10a53273-5b35-4b72-a946-7253763e3168\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"
Nov 01 06:27:29.149455 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:29.149301    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a53273-5b35-4b72-a946-7253763e3168-openshift-service-ca\") pod \"maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f\" (UID: \"10a53273-5b35-4b72-a946-7253763e3168\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"
Nov 01 06:27:29.149455 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:29.149352    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxwbp\" (UniqueName: \"kubernetes.io/projected/10a53273-5b35-4b72-a946-7253763e3168-kube-api-access-zxwbp\") pod \"maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f\" (UID: \"10a53273-5b35-4b72-a946-7253763e3168\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"
Nov 01 06:27:29.149455 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:29.149415    5619 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-serving-cert: secret "maas-api-e2e-ait-ce346503-serving-cert" not found
Nov 01 06:27:29.149455 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:29.149445    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a53273-5b35-4b72-a946-7253763e3168-metrics-certs\") pod \"maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f\" (UID: \"10a53273-5b35-4b72-a946-7253763e3168\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"
Nov 01 06:27:29.149608 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:29.149495    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a53273-5b35-4b72-a946-7253763e3168-maas-api-tls podName:10a53273-5b35-4b72-a946-7253763e3168 nodeName:}" failed. No retries permitted until 2026-11-01 06:27:29.649473018 +0000 UTC m=+2486.860296518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/10a53273-5b35-4b72-a946-7253763e3168-maas-api-tls") pod "maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f" (UID: "10a53273-5b35-4b72-a946-7253763e3168") : secret "maas-api-e2e-ait-ce346503-serving-cert" not found
Nov 01 06:27:29.150197 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:29.150169    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a53273-5b35-4b72-a946-7253763e3168-openshift-service-ca\") pod \"maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f\" (UID: \"10a53273-5b35-4b72-a946-7253763e3168\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"
Nov 01 06:27:29.152248 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:29.152216    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a53273-5b35-4b72-a946-7253763e3168-metrics-certs\") pod \"maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f\" (UID: \"10a53273-5b35-4b72-a946-7253763e3168\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"
Nov 01 06:27:29.211848 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:29.211815    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxwbp\" (UniqueName: \"kubernetes.io/projected/10a53273-5b35-4b72-a946-7253763e3168-kube-api-access-zxwbp\") pod \"maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f\" (UID: \"10a53273-5b35-4b72-a946-7253763e3168\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"
Nov 01 06:27:29.220243 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:29.220208    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:29.405646 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:29.405604    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:29.656483 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:29.656405    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10a53273-5b35-4b72-a946-7253763e3168-maas-api-tls\") pod \"maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f\" (UID: \"10a53273-5b35-4b72-a946-7253763e3168\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"
Nov 01 06:27:29.658749 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:29.658728    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10a53273-5b35-4b72-a946-7253763e3168-maas-api-tls\") pod \"maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f\" (UID: \"10a53273-5b35-4b72-a946-7253763e3168\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"
Nov 01 06:27:29.950579 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:29.950498    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"
Nov 01 06:27:30.326515 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:30.326488    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"]
Nov 01 06:27:30.327879 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:27:30.327856    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a53273_5b35_4b72_a946_7253763e3168.slice/crio-df8dd801ef6b436f732db4c527d3c265a77810f1383155906173e20bd5bf7f30 WatchSource:0}: Error finding container df8dd801ef6b436f732db4c527d3c265a77810f1383155906173e20bd5bf7f30: Status 404 returned error can't find the container with id df8dd801ef6b436f732db4c527d3c265a77810f1383155906173e20bd5bf7f30
Nov 01 06:27:30.407887 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:30.407858    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:30.568257 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:30.568231    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-47cacd68a67a: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:27:30.568257 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:30.568253    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:27:30.568502 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:30.568300    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials podName:8c54c622-970a-4d9a-8bb6-06803bcf1a42 nodeName:}" failed. No retries permitted until 2026-11-01 06:27:46.568287253 +0000 UTC m=+2503.779110741 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7" (UID: "8c54c622-970a-4d9a-8bb6-06803bcf1a42") : secret "agc-cred-47cacd68a67a" not found
Nov 01 06:27:30.907642 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:30.907606    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f" event={"ID":"10a53273-5b35-4b72-a946-7253763e3168","Type":"ContainerStarted","Data":"df8dd801ef6b436f732db4c527d3c265a77810f1383155906173e20bd5bf7f30"}
Nov 01 06:27:31.916273 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:31.916236    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f" event={"ID":"10a53273-5b35-4b72-a946-7253763e3168","Type":"ContainerStarted","Data":"b7590f28b2c9364a35d779549ab5e296e9b2b047aa6ebc55cacbdb27403babee"}
Nov 01 06:27:31.916666 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:31.916306    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"
Nov 01 06:27:31.958759 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:31.958729    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:32.037792 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:32.037737    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f" podStartSLOduration=3.209956409 podStartE2EDuration="4.037723799s" totalImagesPullingTime="827.76739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:27:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:27:32.034895889 +0000 UTC m=+2489.245719398" watchObservedRunningTime="2026-11-01 06:27:32.037723799 +0000 UTC m=+2489.248547307"
Nov 01 06:27:32.122833 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:32.122797    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:32.272406 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:32.272291    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:32.800459 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:32.800417    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:27:34.481081 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:34.481033    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:35.102343 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:35.099715    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:35.246696 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:35.246663    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:36.134363 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:36.134317    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"]
Nov 01 06:27:36.288104 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:36.288063    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:37.472585 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:37.472542    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:37.670604 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:37.670570    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:37.930412 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:37.930373    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"
Nov 01 06:27:38.074547 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:38.074508    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:39.471355 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:39.471304    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:40.164616 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:40.164573    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:40.360977 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:40.360932    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:41.135613 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:41.135575    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59" podUID="3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa" containerName="payload-processing" containerID="cri-o://8fc31e2467493ae56205df9735c9333fb0b328cbd6a1048ae210e1c97fd31cbd" gracePeriod=25
Nov 01 06:27:41.410586 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:41.410559    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"
Nov 01 06:27:41.530735 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:41.530703    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-kube-api-access-dpcbk\" (UniqueName: \"kubernetes.io/projected/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-kube-api-access-dpcbk\") pod \"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa\" (UID: \"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa\") "
Nov 01 06:27:41.531113 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:41.530780    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-provider-credentials\") pod \"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa\" (UID: \"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa\") "
Nov 01 06:27:41.531113 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:41.530807    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-plugins-config-volume\") pod \"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa\" (UID: \"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa\") "
Nov 01 06:27:41.531113 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:41.530971    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-routing-overlay-volume\") pod \"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa\" (UID: \"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa\") "
Nov 01 06:27:41.531765 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:41.531729    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-routing-overlay-volume" pod "3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa" (UID: "3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:27:41.532529 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:41.532167    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-plugins-config-volume" pod "3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa" (UID: "3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:27:41.538862 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:41.538834    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-kube-api-access-dpcbk" pod "3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa" (UID: "3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa"). InnerVolumeSpecName "kube-api-access-dpcbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:27:41.538997 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:41.538856    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-provider-credentials" pod "3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa" (UID: "3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:27:41.632482 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:41.632450    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:27:41.632482 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:41.632476    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpcbk\" (UniqueName: \"kubernetes.io/projected/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-kube-api-access-dpcbk\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:27:41.632482 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:41.632487    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:27:41.632697 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:41.632495    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:27:42.001279 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:42.001237    5619 generic.go:356] "Generic (PLEG): container finished" podID="3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa" containerID="8fc31e2467493ae56205df9735c9333fb0b328cbd6a1048ae210e1c97fd31cbd" exitCode=0
Nov 01 06:27:42.001470 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:42.001287    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"
Nov 01 06:27:42.001470 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:42.001344    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59" event={"ID":"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa","Type":"ContainerDied","Data":"8fc31e2467493ae56205df9735c9333fb0b328cbd6a1048ae210e1c97fd31cbd"}
Nov 01 06:27:42.001470 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:42.001404    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59" event={"ID":"3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa","Type":"ContainerDied","Data":"46ef663300bacb0157869c90ee8215f424d918ae6d22c289512b6115f4ec1090"}
Nov 01 06:27:42.001470 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:42.001427    5619 scope.go:122] "RemoveContainer" containerID="8fc31e2467493ae56205df9735c9333fb0b328cbd6a1048ae210e1c97fd31cbd"
Nov 01 06:27:42.011508 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:42.011490    5619 scope.go:122] "RemoveContainer" containerID="8fc31e2467493ae56205df9735c9333fb0b328cbd6a1048ae210e1c97fd31cbd"
Nov 01 06:27:42.011790 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:42.011768    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fc31e2467493ae56205df9735c9333fb0b328cbd6a1048ae210e1c97fd31cbd\": container with ID starting with 8fc31e2467493ae56205df9735c9333fb0b328cbd6a1048ae210e1c97fd31cbd not found: ID does not exist" containerID="8fc31e2467493ae56205df9735c9333fb0b328cbd6a1048ae210e1c97fd31cbd"
Nov 01 06:27:42.011867 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:42.011798    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc31e2467493ae56205df9735c9333fb0b328cbd6a1048ae210e1c97fd31cbd"} err="failed to get container status \"8fc31e2467493ae56205df9735c9333fb0b328cbd6a1048ae210e1c97fd31cbd\": rpc error: code = NotFound desc = could not find container \"8fc31e2467493ae56205df9735c9333fb0b328cbd6a1048ae210e1c97fd31cbd\": container with ID starting with 8fc31e2467493ae56205df9735c9333fb0b328cbd6a1048ae210e1c97fd31cbd not found: ID does not exist"
Nov 01 06:27:42.169392 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:42.169359    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"]
Nov 01 06:27:42.219888 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:42.219857    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-59f5fcbf6-tdc59"]
Nov 01 06:27:42.542139 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:42.542096    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:42.801385 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:42.801288    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:27:42.966919 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:42.966887    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa" path="/var/lib/kubelet/pods/3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa/volumes"
Nov 01 06:27:43.661624 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.661582    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:43.705345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.698374    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"]
Nov 01 06:27:43.705345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.699731    5619 cpu_manager.go:414] "containerMap: removing container" podUID="3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa" containerName="payload-processing"
Nov 01 06:27:43.705345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.699751    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa" containerName="payload-processing"
Nov 01 06:27:43.705345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.700069    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d1900c0-9e4b-43b1-b2a9-9b017f3cd1aa" containerName="payload-processing"
Nov 01 06:27:43.738347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.737821    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"
Nov 01 06:27:43.762283 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.762248    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"]
Nov 01 06:27:43.808693 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.808665    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1990a5ad24d9\""
Nov 01 06:27:43.808907 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.808731    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 06:27:43.808907 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.808779    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 06:27:43.808907 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.808785    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 06:27:43.809090 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.808885    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-6c7jv\""
Nov 01 06:27:43.842623 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.841846    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 06:27:43.850137 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.850097    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:43.862878 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.862841    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/543ed64e-005b-4c2f-9c85-21df71b51900-routing-overlay-volume\") pod \"payload-processing-external-model-65f758f5bc-sdxxv\" (UID: \"543ed64e-005b-4c2f-9c85-21df71b51900\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"
Nov 01 06:27:43.863125 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.862900    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlbc5\" (UniqueName: \"kubernetes.io/projected/543ed64e-005b-4c2f-9c85-21df71b51900-kube-api-access-xlbc5\") pod \"payload-processing-external-model-65f758f5bc-sdxxv\" (UID: \"543ed64e-005b-4c2f-9c85-21df71b51900\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"
Nov 01 06:27:43.863125 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.863020    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543ed64e-005b-4c2f-9c85-21df71b51900-plugins-config-volume\") pod \"payload-processing-external-model-65f758f5bc-sdxxv\" (UID: \"543ed64e-005b-4c2f-9c85-21df71b51900\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"
Nov 01 06:27:43.863308 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.863123    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/543ed64e-005b-4c2f-9c85-21df71b51900-provider-credentials\") pod \"payload-processing-external-model-65f758f5bc-sdxxv\" (UID: \"543ed64e-005b-4c2f-9c85-21df71b51900\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"
Nov 01 06:27:43.963860 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.963777    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543ed64e-005b-4c2f-9c85-21df71b51900-plugins-config-volume\") pod \"payload-processing-external-model-65f758f5bc-sdxxv\" (UID: \"543ed64e-005b-4c2f-9c85-21df71b51900\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"
Nov 01 06:27:43.964016 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.963872    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/543ed64e-005b-4c2f-9c85-21df71b51900-provider-credentials\") pod \"payload-processing-external-model-65f758f5bc-sdxxv\" (UID: \"543ed64e-005b-4c2f-9c85-21df71b51900\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"
Nov 01 06:27:43.964016 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:43.963963    5619 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1990a5ad24d9: secret "agc-cred-1990a5ad24d9" not found
Nov 01 06:27:43.964016 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:43.963983    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv: secret "agc-cred-1990a5ad24d9" not found
Nov 01 06:27:43.964016 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.964005    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/543ed64e-005b-4c2f-9c85-21df71b51900-routing-overlay-volume\") pod \"payload-processing-external-model-65f758f5bc-sdxxv\" (UID: \"543ed64e-005b-4c2f-9c85-21df71b51900\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"
Nov 01 06:27:43.964220 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.964030    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlbc5\" (UniqueName: \"kubernetes.io/projected/543ed64e-005b-4c2f-9c85-21df71b51900-kube-api-access-xlbc5\") pod \"payload-processing-external-model-65f758f5bc-sdxxv\" (UID: \"543ed64e-005b-4c2f-9c85-21df71b51900\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"
Nov 01 06:27:43.964220 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:43.964051    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/543ed64e-005b-4c2f-9c85-21df71b51900-provider-credentials podName:543ed64e-005b-4c2f-9c85-21df71b51900 nodeName:}" failed. No retries permitted until 2026-11-01 06:27:44.464032364 +0000 UTC m=+2501.674855857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/543ed64e-005b-4c2f-9c85-21df71b51900-provider-credentials") pod "payload-processing-external-model-65f758f5bc-sdxxv" (UID: "543ed64e-005b-4c2f-9c85-21df71b51900") : secret "agc-cred-1990a5ad24d9" not found
Nov 01 06:27:43.964637 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.964613    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543ed64e-005b-4c2f-9c85-21df71b51900-plugins-config-volume\") pod \"payload-processing-external-model-65f758f5bc-sdxxv\" (UID: \"543ed64e-005b-4c2f-9c85-21df71b51900\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"
Nov 01 06:27:43.964733 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:43.964695    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/543ed64e-005b-4c2f-9c85-21df71b51900-routing-overlay-volume\") pod \"payload-processing-external-model-65f758f5bc-sdxxv\" (UID: \"543ed64e-005b-4c2f-9c85-21df71b51900\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"
Nov 01 06:27:44.026418 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:44.026388    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlbc5\" (UniqueName: \"kubernetes.io/projected/543ed64e-005b-4c2f-9c85-21df71b51900-kube-api-access-xlbc5\") pod \"payload-processing-external-model-65f758f5bc-sdxxv\" (UID: \"543ed64e-005b-4c2f-9c85-21df71b51900\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"
Nov 01 06:27:44.449453 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:44.449416    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:44.470468 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:44.470435    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/543ed64e-005b-4c2f-9c85-21df71b51900-provider-credentials\") pod \"payload-processing-external-model-65f758f5bc-sdxxv\" (UID: \"543ed64e-005b-4c2f-9c85-21df71b51900\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"
Nov 01 06:27:44.470609 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:44.470562    5619 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1990a5ad24d9: secret "agc-cred-1990a5ad24d9" not found
Nov 01 06:27:44.470609 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:44.470579    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv: secret "agc-cred-1990a5ad24d9" not found
Nov 01 06:27:44.470706 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:44.470630    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/543ed64e-005b-4c2f-9c85-21df71b51900-provider-credentials podName:543ed64e-005b-4c2f-9c85-21df71b51900 nodeName:}" failed. No retries permitted until 2026-11-01 06:27:45.470616722 +0000 UTC m=+2502.681440227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/543ed64e-005b-4c2f-9c85-21df71b51900-provider-credentials") pod "payload-processing-external-model-65f758f5bc-sdxxv" (UID: "543ed64e-005b-4c2f-9c85-21df71b51900") : secret "agc-cred-1990a5ad24d9" not found
Nov 01 06:27:45.482425 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:45.482393    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/543ed64e-005b-4c2f-9c85-21df71b51900-provider-credentials\") pod \"payload-processing-external-model-65f758f5bc-sdxxv\" (UID: \"543ed64e-005b-4c2f-9c85-21df71b51900\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"
Nov 01 06:27:45.482857 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:45.482537    5619 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1990a5ad24d9: secret "agc-cred-1990a5ad24d9" not found
Nov 01 06:27:45.482857 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:45.482553    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv: secret "agc-cred-1990a5ad24d9" not found
Nov 01 06:27:45.482857 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:45.482607    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/543ed64e-005b-4c2f-9c85-21df71b51900-provider-credentials podName:543ed64e-005b-4c2f-9c85-21df71b51900 nodeName:}" failed. No retries permitted until 2026-11-01 06:27:47.482592481 +0000 UTC m=+2504.693415972 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/543ed64e-005b-4c2f-9c85-21df71b51900-provider-credentials") pod "payload-processing-external-model-65f758f5bc-sdxxv" (UID: "543ed64e-005b-4c2f-9c85-21df71b51900") : secret "agc-cred-1990a5ad24d9" not found
Nov 01 06:27:46.190669 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:46.190632    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"]
Nov 01 06:27:46.190946 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:46.190926    5619 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv" podUID="543ed64e-005b-4c2f-9c85-21df71b51900"
Nov 01 06:27:46.329217 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:46.329177    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:46.594952 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:46.594929    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-47cacd68a67a: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:27:46.594952 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:46.594954    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:27:46.595367 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:46.595006    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials podName:8c54c622-970a-4d9a-8bb6-06803bcf1a42 nodeName:}" failed. No retries permitted until 2026-11-01 06:28:18.594989232 +0000 UTC m=+2535.805812723 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7" (UID: "8c54c622-970a-4d9a-8bb6-06803bcf1a42") : secret "agc-cred-47cacd68a67a" not found
Nov 01 06:27:47.047406 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:47.046435    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"
Nov 01 06:27:47.058907 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:47.058869    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"
Nov 01 06:27:47.108400 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:47.108369    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/543ed64e-005b-4c2f-9c85-21df71b51900-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/543ed64e-005b-4c2f-9c85-21df71b51900-routing-overlay-volume\") pod \"543ed64e-005b-4c2f-9c85-21df71b51900\" (UID: \"543ed64e-005b-4c2f-9c85-21df71b51900\") "
Nov 01 06:27:47.108589 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:47.108482    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/543ed64e-005b-4c2f-9c85-21df71b51900-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543ed64e-005b-4c2f-9c85-21df71b51900-plugins-config-volume\") pod \"543ed64e-005b-4c2f-9c85-21df71b51900\" (UID: \"543ed64e-005b-4c2f-9c85-21df71b51900\") "
Nov 01 06:27:47.108589 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:47.108539    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/543ed64e-005b-4c2f-9c85-21df71b51900-kube-api-access-xlbc5\" (UniqueName: \"kubernetes.io/projected/543ed64e-005b-4c2f-9c85-21df71b51900-kube-api-access-xlbc5\") pod \"543ed64e-005b-4c2f-9c85-21df71b51900\" (UID: \"543ed64e-005b-4c2f-9c85-21df71b51900\") "
Nov 01 06:27:47.108800 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:47.108773    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/543ed64e-005b-4c2f-9c85-21df71b51900-routing-overlay-volume" pod "543ed64e-005b-4c2f-9c85-21df71b51900" (UID: "543ed64e-005b-4c2f-9c85-21df71b51900"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:27:47.108920 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:47.108898    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/543ed64e-005b-4c2f-9c85-21df71b51900-plugins-config-volume" pod "543ed64e-005b-4c2f-9c85-21df71b51900" (UID: "543ed64e-005b-4c2f-9c85-21df71b51900"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:27:47.109112 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:47.109094    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543ed64e-005b-4c2f-9c85-21df71b51900-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:27:47.109176 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:47.109114    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/543ed64e-005b-4c2f-9c85-21df71b51900-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:27:47.110773 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:47.110748    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543ed64e-005b-4c2f-9c85-21df71b51900-kube-api-access-xlbc5" pod "543ed64e-005b-4c2f-9c85-21df71b51900" (UID: "543ed64e-005b-4c2f-9c85-21df71b51900"). InnerVolumeSpecName "kube-api-access-xlbc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:27:47.210803 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:47.210764    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlbc5\" (UniqueName: \"kubernetes.io/projected/543ed64e-005b-4c2f-9c85-21df71b51900-kube-api-access-xlbc5\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:27:47.515149 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:47.515112    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/543ed64e-005b-4c2f-9c85-21df71b51900-provider-credentials\") pod \"payload-processing-external-model-65f758f5bc-sdxxv\" (UID: \"543ed64e-005b-4c2f-9c85-21df71b51900\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"
Nov 01 06:27:47.515360 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:47.515266    5619 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1990a5ad24d9: secret "agc-cred-1990a5ad24d9" not found
Nov 01 06:27:47.515360 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:47.515282    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv: secret "agc-cred-1990a5ad24d9" not found
Nov 01 06:27:47.515360 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:47.515358    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/543ed64e-005b-4c2f-9c85-21df71b51900-provider-credentials podName:543ed64e-005b-4c2f-9c85-21df71b51900 nodeName:}" failed. No retries permitted until 2026-11-01 06:27:51.5153418 +0000 UTC m=+2508.726165303 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/543ed64e-005b-4c2f-9c85-21df71b51900-provider-credentials") pod "payload-processing-external-model-65f758f5bc-sdxxv" (UID: "543ed64e-005b-4c2f-9c85-21df71b51900") : secret "agc-cred-1990a5ad24d9" not found
Nov 01 06:27:48.051671 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:48.051626    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"
Nov 01 06:27:48.282304 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:48.282271    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"]
Nov 01 06:27:48.286475 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:48.286444    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-65f758f5bc-sdxxv"]
Nov 01 06:27:48.324706 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:48.324637    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/543ed64e-005b-4c2f-9c85-21df71b51900-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:27:48.962098 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:48.962066    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="543ed64e-005b-4c2f-9c85-21df71b51900" path="/var/lib/kubelet/pods/543ed64e-005b-4c2f-9c85-21df71b51900/volumes"
Nov 01 06:27:49.273736 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:49.273652    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:49.852035 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:49.851997    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:50.046568 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:50.046518    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:52.370522 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:52.370489    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:52.801097 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:52.801003    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:27:53.530966 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:53.530933    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"]
Nov 01 06:27:53.535389 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:53.535370    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"
Nov 01 06:27:53.590341 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:53.590271    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1990a5ad24d9\""
Nov 01 06:27:53.599011 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:53.598974    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8a51161-c414-4601-ba48-893c15fd09a4-routing-overlay-volume\") pod \"payload-processing-external-model-65f758f5bc-svcjd\" (UID: \"b8a51161-c414-4601-ba48-893c15fd09a4\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"
Nov 01 06:27:53.599174 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:53.599068    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8a51161-c414-4601-ba48-893c15fd09a4-provider-credentials\") pod \"payload-processing-external-model-65f758f5bc-svcjd\" (UID: \"b8a51161-c414-4601-ba48-893c15fd09a4\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"
Nov 01 06:27:53.599174 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:53.599113    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8a51161-c414-4601-ba48-893c15fd09a4-plugins-config-volume\") pod \"payload-processing-external-model-65f758f5bc-svcjd\" (UID: \"b8a51161-c414-4601-ba48-893c15fd09a4\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"
Nov 01 06:27:53.599174 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:53.599170    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnrhm\" (UniqueName: \"kubernetes.io/projected/b8a51161-c414-4601-ba48-893c15fd09a4-kube-api-access-jnrhm\") pod \"payload-processing-external-model-65f758f5bc-svcjd\" (UID: \"b8a51161-c414-4601-ba48-893c15fd09a4\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"
Nov 01 06:27:53.641113 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:53.641068    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 06:27:53.641113 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:53.641087    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 06:27:53.641113 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:53.641098    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 06:27:53.641356 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:53.641081    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 06:27:53.641356 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:53.641068    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-p5k62\""
Nov 01 06:27:53.645511 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:53.645489    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"]
Nov 01 06:27:53.700740 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:53.700697    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8a51161-c414-4601-ba48-893c15fd09a4-routing-overlay-volume\") pod \"payload-processing-external-model-65f758f5bc-svcjd\" (UID: \"b8a51161-c414-4601-ba48-893c15fd09a4\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"
Nov 01 06:27:53.700984 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:53.700769    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8a51161-c414-4601-ba48-893c15fd09a4-provider-credentials\") pod \"payload-processing-external-model-65f758f5bc-svcjd\" (UID: \"b8a51161-c414-4601-ba48-893c15fd09a4\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"
Nov 01 06:27:53.700984 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:53.700796    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8a51161-c414-4601-ba48-893c15fd09a4-plugins-config-volume\") pod \"payload-processing-external-model-65f758f5bc-svcjd\" (UID: \"b8a51161-c414-4601-ba48-893c15fd09a4\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"
Nov 01 06:27:53.700984 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:53.700834    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnrhm\" (UniqueName: \"kubernetes.io/projected/b8a51161-c414-4601-ba48-893c15fd09a4-kube-api-access-jnrhm\") pod \"payload-processing-external-model-65f758f5bc-svcjd\" (UID: \"b8a51161-c414-4601-ba48-893c15fd09a4\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"
Nov 01 06:27:53.701440 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:53.701413    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8a51161-c414-4601-ba48-893c15fd09a4-routing-overlay-volume\") pod \"payload-processing-external-model-65f758f5bc-svcjd\" (UID: \"b8a51161-c414-4601-ba48-893c15fd09a4\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"
Nov 01 06:27:53.701746 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:53.701724    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8a51161-c414-4601-ba48-893c15fd09a4-plugins-config-volume\") pod \"payload-processing-external-model-65f758f5bc-svcjd\" (UID: \"b8a51161-c414-4601-ba48-893c15fd09a4\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"
Nov 01 06:27:53.703872 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:53.703848    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8a51161-c414-4601-ba48-893c15fd09a4-provider-credentials\") pod \"payload-processing-external-model-65f758f5bc-svcjd\" (UID: \"b8a51161-c414-4601-ba48-893c15fd09a4\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"
Nov 01 06:27:53.762730 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:53.762703    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnrhm\" (UniqueName: \"kubernetes.io/projected/b8a51161-c414-4601-ba48-893c15fd09a4-kube-api-access-jnrhm\") pod \"payload-processing-external-model-65f758f5bc-svcjd\" (UID: \"b8a51161-c414-4601-ba48-893c15fd09a4\") " pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"
Nov 01 06:27:53.851070 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:53.851043    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"
Nov 01 06:27:54.060202 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:54.060168    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:54.235613 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:54.235576    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"]
Nov 01 06:27:54.243078 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:27:54.243047    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a51161_c414_4601_ba48_893c15fd09a4.slice/crio-3a87d614a9a3ef0f5d46572758b2c3038557d500533c81a24664a39aa1da9811 WatchSource:0}: Error finding container 3a87d614a9a3ef0f5d46572758b2c3038557d500533c81a24664a39aa1da9811: Status 404 returned error can't find the container with id 3a87d614a9a3ef0f5d46572758b2c3038557d500533c81a24664a39aa1da9811
Nov 01 06:27:55.095560 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:55.095526    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd" event={"ID":"b8a51161-c414-4601-ba48-893c15fd09a4","Type":"ContainerStarted","Data":"bdc08b42c04c8698c53d1e241b2202b93dad942c67894e5072c5388abb8c3a11"}
Nov 01 06:27:55.095975 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:55.095584    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd" event={"ID":"b8a51161-c414-4601-ba48-893c15fd09a4","Type":"ContainerStarted","Data":"3a87d614a9a3ef0f5d46572758b2c3038557d500533c81a24664a39aa1da9811"}
Nov 01 06:27:55.095975 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:55.095733    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"
Nov 01 06:27:55.214106 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:55.214055    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd" podStartSLOduration=2.21404084 podStartE2EDuration="2.21404084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:27:55.213108294 +0000 UTC m=+2512.423931802" watchObservedRunningTime="2026-11-01 06:27:55.21404084 +0000 UTC m=+2512.424864380"
Nov 01 06:27:55.880981 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:27:55.880940    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:27:56.472756 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:56.472715    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:56.501467 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:56.501427    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"]
Nov 01 06:27:56.631564 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:56.631534    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:56.833107 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:56.833013    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:27:58.826315 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:27:58.826280    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:02.110448 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:02.110394    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd" podUID="b8a51161-c414-4601-ba48-893c15fd09a4" containerName="payload-processing" containerID="cri-o://bdc08b42c04c8698c53d1e241b2202b93dad942c67894e5072c5388abb8c3a11" gracePeriod=25
Nov 01 06:28:02.665060 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:02.664702    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"
Nov 01 06:28:02.740459 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:02.740425    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a51161-c414-4601-ba48-893c15fd09a4-kube-api-access-jnrhm\" (UniqueName: \"kubernetes.io/projected/b8a51161-c414-4601-ba48-893c15fd09a4-kube-api-access-jnrhm\") pod \"b8a51161-c414-4601-ba48-893c15fd09a4\" (UID: \"b8a51161-c414-4601-ba48-893c15fd09a4\") "
Nov 01 06:28:02.740645 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:02.740506    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8a51161-c414-4601-ba48-893c15fd09a4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8a51161-c414-4601-ba48-893c15fd09a4-routing-overlay-volume\") pod \"b8a51161-c414-4601-ba48-893c15fd09a4\" (UID: \"b8a51161-c414-4601-ba48-893c15fd09a4\") "
Nov 01 06:28:02.740645 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:02.740576    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a51161-c414-4601-ba48-893c15fd09a4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8a51161-c414-4601-ba48-893c15fd09a4-provider-credentials\") pod \"b8a51161-c414-4601-ba48-893c15fd09a4\" (UID: \"b8a51161-c414-4601-ba48-893c15fd09a4\") "
Nov 01 06:28:02.740645 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:02.740602    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8a51161-c414-4601-ba48-893c15fd09a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8a51161-c414-4601-ba48-893c15fd09a4-plugins-config-volume\") pod \"b8a51161-c414-4601-ba48-893c15fd09a4\" (UID: \"b8a51161-c414-4601-ba48-893c15fd09a4\") "
Nov 01 06:28:02.740928 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:02.740854    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8a51161-c414-4601-ba48-893c15fd09a4-routing-overlay-volume" pod "b8a51161-c414-4601-ba48-893c15fd09a4" (UID: "b8a51161-c414-4601-ba48-893c15fd09a4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:28:02.741067 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:02.741018    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8a51161-c414-4601-ba48-893c15fd09a4-plugins-config-volume" pod "b8a51161-c414-4601-ba48-893c15fd09a4" (UID: "b8a51161-c414-4601-ba48-893c15fd09a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:28:02.741128 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:02.741109    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8a51161-c414-4601-ba48-893c15fd09a4-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:02.742634 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:02.742613    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a51161-c414-4601-ba48-893c15fd09a4-kube-api-access-jnrhm" pod "b8a51161-c414-4601-ba48-893c15fd09a4" (UID: "b8a51161-c414-4601-ba48-893c15fd09a4"). InnerVolumeSpecName "kube-api-access-jnrhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:28:02.742767 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:02.742747    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a51161-c414-4601-ba48-893c15fd09a4-provider-credentials" pod "b8a51161-c414-4601-ba48-893c15fd09a4" (UID: "b8a51161-c414-4601-ba48-893c15fd09a4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:28:02.835074 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:02.830300    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:28:02.846234 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:02.843112    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8a51161-c414-4601-ba48-893c15fd09a4-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:02.846234 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:02.843149    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8a51161-c414-4601-ba48-893c15fd09a4-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:02.846234 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:02.843165    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnrhm\" (UniqueName: \"kubernetes.io/projected/b8a51161-c414-4601-ba48-893c15fd09a4-kube-api-access-jnrhm\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:28:03.148902 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:03.148869    5619 generic.go:356] "Generic (PLEG): container finished" podID="b8a51161-c414-4601-ba48-893c15fd09a4" containerID="bdc08b42c04c8698c53d1e241b2202b93dad942c67894e5072c5388abb8c3a11" exitCode=0
Nov 01 06:28:03.148902 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:03.148908    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd" event={"ID":"b8a51161-c414-4601-ba48-893c15fd09a4","Type":"ContainerDied","Data":"bdc08b42c04c8698c53d1e241b2202b93dad942c67894e5072c5388abb8c3a11"}
Nov 01 06:28:03.149413 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:03.148915    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"
Nov 01 06:28:03.149413 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:03.148949    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd" event={"ID":"b8a51161-c414-4601-ba48-893c15fd09a4","Type":"ContainerDied","Data":"3a87d614a9a3ef0f5d46572758b2c3038557d500533c81a24664a39aa1da9811"}
Nov 01 06:28:03.149413 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:03.148971    5619 scope.go:122] "RemoveContainer" containerID="bdc08b42c04c8698c53d1e241b2202b93dad942c67894e5072c5388abb8c3a11"
Nov 01 06:28:03.158258 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:03.158240    5619 scope.go:122] "RemoveContainer" containerID="bdc08b42c04c8698c53d1e241b2202b93dad942c67894e5072c5388abb8c3a11"
Nov 01 06:28:03.158546 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:28:03.158525    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc08b42c04c8698c53d1e241b2202b93dad942c67894e5072c5388abb8c3a11\": container with ID starting with bdc08b42c04c8698c53d1e241b2202b93dad942c67894e5072c5388abb8c3a11 not found: ID does not exist" containerID="bdc08b42c04c8698c53d1e241b2202b93dad942c67894e5072c5388abb8c3a11"
Nov 01 06:28:03.158611 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:03.158554    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc08b42c04c8698c53d1e241b2202b93dad942c67894e5072c5388abb8c3a11"} err="failed to get container status \"bdc08b42c04c8698c53d1e241b2202b93dad942c67894e5072c5388abb8c3a11\": rpc error: code = NotFound desc = could not find container \"bdc08b42c04c8698c53d1e241b2202b93dad942c67894e5072c5388abb8c3a11\": container with ID starting with bdc08b42c04c8698c53d1e241b2202b93dad942c67894e5072c5388abb8c3a11 not found: ID does not exist"
Nov 01 06:28:03.316230 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:03.316150    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"]
Nov 01 06:28:03.319948 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:03.319923    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-65f758f5bc-svcjd"]
Nov 01 06:28:03.663447 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:03.663414    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:04.962399 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:04.962368    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8a51161-c414-4601-ba48-893c15fd09a4" path="/var/lib/kubelet/pods/b8a51161-c414-4601-ba48-893c15fd09a4/volumes"
Nov 01 06:28:08.409782 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:08.409739    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:08.599044 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:08.599009    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:08.738506 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:08.738416    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:12.056620 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:12.056415    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:12.248901 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:12.248866    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:12.682968 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:12.682929    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:12.800457 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:12.800418    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:28:13.154744 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:13.154710    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:13.956422 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:13.956380    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:28:15.192858 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:15.192822    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:15.333509 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:15.333454    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:16.759934 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:16.759889    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:17.957293 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:17.957253    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:18.634880 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:28:18.634847    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-47cacd68a67a: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:28:18.634880 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:28:18.634872    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:28:18.635086 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:28:18.634926    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials podName:8c54c622-970a-4d9a-8bb6-06803bcf1a42 nodeName:}" failed. No retries permitted until 2026-11-01 06:29:22.634912113 +0000 UTC m=+2599.845735601 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7" (UID: "8c54c622-970a-4d9a-8bb6-06803bcf1a42") : secret "agc-cred-47cacd68a67a" not found
Nov 01 06:28:19.959480 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:19.959443    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:20.149594 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:20.149544    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:22.675173 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:22.675133    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:23.956276 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:23.956238    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:28:25.879084 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:28:25.879045    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:28:26.033555 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:26.033509    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:27.374462 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:27.374426    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:28.207783 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:28.207746    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:29.336100 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:29.336065    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:29.534433 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:29.534395    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:30.752148 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:30.752110    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:32.034406 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:32.034372    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:32.225374 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:32.225308    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:34.001346 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:33.990413    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:28:34.532226 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:34.532189    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:34.732350 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:34.731155    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:34.890347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:34.889974    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:37.462067 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:37.462032    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:38.940613 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:38.940575    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:39.117900 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.117864    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:39.358541 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.358508    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9"]
Nov 01 06:28:39.359427 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.359408    5619 cpu_manager.go:414] "containerMap: removing container" podUID="b8a51161-c414-4601-ba48-893c15fd09a4" containerName="payload-processing"
Nov 01 06:28:39.359482 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.359431    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8a51161-c414-4601-ba48-893c15fd09a4" containerName="payload-processing"
Nov 01 06:28:39.359590 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.359579    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8a51161-c414-4601-ba48-893c15fd09a4" containerName="payload-processing"
Nov 01 06:28:39.363514 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.363494    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9"
Nov 01 06:28:39.417135 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.417106    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s5zj\" (UniqueName: \"kubernetes.io/projected/7f4f13d7-922b-4478-afe8-0b9c44d894ce-kube-api-access-4s5zj\") pod \"payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9\" (UID: \"7f4f13d7-922b-4478-afe8-0b9c44d894ce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9"
Nov 01 06:28:39.417537 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.417498    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f4f13d7-922b-4478-afe8-0b9c44d894ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9\" (UID: \"7f4f13d7-922b-4478-afe8-0b9c44d894ce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9"
Nov 01 06:28:39.419381 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.419361    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9"]
Nov 01 06:28:39.485539 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.485502    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-d7d444\""
Nov 01 06:28:39.489037 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.488996    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-d7d444-dockercfg-fr8xs\""
Nov 01 06:28:39.509400 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.509361    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4"]
Nov 01 06:28:39.515116 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.514574    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4"
Nov 01 06:28:39.519503 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.519470    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s5zj\" (UniqueName: \"kubernetes.io/projected/7f4f13d7-922b-4478-afe8-0b9c44d894ce-kube-api-access-4s5zj\") pod \"payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9\" (UID: \"7f4f13d7-922b-4478-afe8-0b9c44d894ce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9"
Nov 01 06:28:39.519660 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.519640    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f4f13d7-922b-4478-afe8-0b9c44d894ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9\" (UID: \"7f4f13d7-922b-4478-afe8-0b9c44d894ce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9"
Nov 01 06:28:39.520656 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.520626    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f4f13d7-922b-4478-afe8-0b9c44d894ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9\" (UID: \"7f4f13d7-922b-4478-afe8-0b9c44d894ce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9"
Nov 01 06:28:39.575354 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.573802    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4"]
Nov 01 06:28:39.584094 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.584067    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s5zj\" (UniqueName: \"kubernetes.io/projected/7f4f13d7-922b-4478-afe8-0b9c44d894ce-kube-api-access-4s5zj\") pod \"payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9\" (UID: \"7f4f13d7-922b-4478-afe8-0b9c44d894ce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9"
Nov 01 06:28:39.621131 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.621054    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b15032-f506-448c-a9a6-f5f45930f3a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4\" (UID: \"a8b15032-f506-448c-a9a6-f5f45930f3a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4"
Nov 01 06:28:39.621288 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.621152    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxdr5\" (UniqueName: \"kubernetes.io/projected/a8b15032-f506-448c-a9a6-f5f45930f3a6-kube-api-access-hxdr5\") pod \"payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4\" (UID: \"a8b15032-f506-448c-a9a6-f5f45930f3a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4"
Nov 01 06:28:39.678220 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.678184    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9"
Nov 01 06:28:39.722040 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.721992    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b15032-f506-448c-a9a6-f5f45930f3a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4\" (UID: \"a8b15032-f506-448c-a9a6-f5f45930f3a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4"
Nov 01 06:28:39.722224 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.722067    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxdr5\" (UniqueName: \"kubernetes.io/projected/a8b15032-f506-448c-a9a6-f5f45930f3a6-kube-api-access-hxdr5\") pod \"payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4\" (UID: \"a8b15032-f506-448c-a9a6-f5f45930f3a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4"
Nov 01 06:28:39.722713 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.722695    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b15032-f506-448c-a9a6-f5f45930f3a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4\" (UID: \"a8b15032-f506-448c-a9a6-f5f45930f3a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4"
Nov 01 06:28:39.785334 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.785286    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxdr5\" (UniqueName: \"kubernetes.io/projected/a8b15032-f506-448c-a9a6-f5f45930f3a6-kube-api-access-hxdr5\") pod \"payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4\" (UID: \"a8b15032-f506-448c-a9a6-f5f45930f3a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4"
Nov 01 06:28:39.846987 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:39.846952    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4"
Nov 01 06:28:40.069276 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:40.069243    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9"]
Nov 01 06:28:40.070724 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:28:40.070695    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4f13d7_922b_4478_afe8_0b9c44d894ce.slice/crio-1cdb0024846199dd915d3804f61c3ef2085c8199ae341cb98b284ca58e5179a2 WatchSource:0}: Error finding container 1cdb0024846199dd915d3804f61c3ef2085c8199ae341cb98b284ca58e5179a2: Status 404 returned error can't find the container with id 1cdb0024846199dd915d3804f61c3ef2085c8199ae341cb98b284ca58e5179a2
Nov 01 06:28:40.223613 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:40.223584    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4"]
Nov 01 06:28:40.224290 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:28:40.224263    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b15032_f506_448c_a9a6_f5f45930f3a6.slice/crio-fed5680e37053de273e657833e20edfad08e8d99d8663e7c22711a309d3dccc8 WatchSource:0}: Error finding container fed5680e37053de273e657833e20edfad08e8d99d8663e7c22711a309d3dccc8: Status 404 returned error can't find the container with id fed5680e37053de273e657833e20edfad08e8d99d8663e7c22711a309d3dccc8
Nov 01 06:28:40.381141 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:40.381102    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4" event={"ID":"a8b15032-f506-448c-a9a6-f5f45930f3a6","Type":"ContainerStarted","Data":"17bbfe362ebb797fa99877cc9cb4e0546ccc217ebc13d96a98e9a4435410750d"}
Nov 01 06:28:40.381293 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:40.381164    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4" event={"ID":"a8b15032-f506-448c-a9a6-f5f45930f3a6","Type":"ContainerStarted","Data":"fed5680e37053de273e657833e20edfad08e8d99d8663e7c22711a309d3dccc8"}
Nov 01 06:28:40.381397 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:40.381331    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4"
Nov 01 06:28:40.382909 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:40.382880    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9" event={"ID":"7f4f13d7-922b-4478-afe8-0b9c44d894ce","Type":"ContainerStarted","Data":"cdd885ad492c30ba322c7563606d8918fe81234355c04c56d8bfb547f8756c39"}
Nov 01 06:28:40.383029 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:40.382925    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9" event={"ID":"7f4f13d7-922b-4478-afe8-0b9c44d894ce","Type":"ContainerStarted","Data":"1cdb0024846199dd915d3804f61c3ef2085c8199ae341cb98b284ca58e5179a2"}
Nov 01 06:28:40.383029 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:40.382950    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9"
Nov 01 06:28:40.500185 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:40.500091    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4" podStartSLOduration=1.500077996 podStartE2EDuration="1.500077996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:28:40.497831864 +0000 UTC m=+2557.708655372" watchObservedRunningTime="2026-11-01 06:28:40.500077996 +0000 UTC m=+2557.710901505"
Nov 01 06:28:40.615411 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:40.615238    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9" podStartSLOduration=1.6152209339999999 podStartE2EDuration="1.615220934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:28:40.614197937 +0000 UTC m=+2557.825021446" watchObservedRunningTime="2026-11-01 06:28:40.615220934 +0000 UTC m=+2557.826044443"
Nov 01 06:28:42.659686 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:42.659640    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:42.846278 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:42.846233    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:43.520720 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:43.520691    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:43.729599 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:43.729567    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:43.956424 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:43.956389    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:28:47.607793 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:47.607755    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:47.796636 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:47.796600    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:47.945178 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:47.945079    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:48.087894 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:48.087858    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:50.607078 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:50.606927    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:50.748155 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:50.748119    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:51.393708 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:51.393680    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4"
Nov 01 06:28:51.393907 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:51.393758    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9"
Nov 01 06:28:52.663193 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:52.663144    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:53.957151 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:53.957110    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:28:54.022045 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.022011    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:54.192073 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.192032    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:28:54.424784 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.424747    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk"]
Nov 01 06:28:54.429894 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.429874    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk"
Nov 01 06:28:54.484442 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.483804    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c0662f25\""
Nov 01 06:28:54.525877 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.525837    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd5afce-97a4-4a99-b303-3e542ef0ddba-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk\" (UID: \"0dd5afce-97a4-4a99-b303-3e542ef0ddba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk"
Nov 01 06:28:54.526054 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.525991    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpcgp\" (UniqueName: \"kubernetes.io/projected/0dd5afce-97a4-4a99-b303-3e542ef0ddba-kube-api-access-bpcgp\") pod \"payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk\" (UID: \"0dd5afce-97a4-4a99-b303-3e542ef0ddba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk"
Nov 01 06:28:54.545030 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.541501    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c0662f25-dockercfg-4grrj\""
Nov 01 06:28:54.579131 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.547465    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk"]
Nov 01 06:28:54.595471 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.595431    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm"]
Nov 01 06:28:54.601736 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.601707    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm"
Nov 01 06:28:54.627541 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.627508    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vngm5\" (UniqueName: \"kubernetes.io/projected/0f2b2a1a-9247-47c1-bf89-65210876c41f-kube-api-access-vngm5\") pod \"payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm\" (UID: \"0f2b2a1a-9247-47c1-bf89-65210876c41f\") " pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm"
Nov 01 06:28:54.627703 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.627563    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpcgp\" (UniqueName: \"kubernetes.io/projected/0dd5afce-97a4-4a99-b303-3e542ef0ddba-kube-api-access-bpcgp\") pod \"payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk\" (UID: \"0dd5afce-97a4-4a99-b303-3e542ef0ddba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk"
Nov 01 06:28:54.627703 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.627624    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f2b2a1a-9247-47c1-bf89-65210876c41f-plugins-config-volume\") pod \"payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm\" (UID: \"0f2b2a1a-9247-47c1-bf89-65210876c41f\") " pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm"
Nov 01 06:28:54.627888 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.627868    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd5afce-97a4-4a99-b303-3e542ef0ddba-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk\" (UID: \"0dd5afce-97a4-4a99-b303-3e542ef0ddba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk"
Nov 01 06:28:54.628529 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.628511    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd5afce-97a4-4a99-b303-3e542ef0ddba-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk\" (UID: \"0dd5afce-97a4-4a99-b303-3e542ef0ddba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk"
Nov 01 06:28:54.651680 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.651653    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm"]
Nov 01 06:28:54.690056 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.689987    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpcgp\" (UniqueName: \"kubernetes.io/projected/0dd5afce-97a4-4a99-b303-3e542ef0ddba-kube-api-access-bpcgp\") pod \"payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk\" (UID: \"0dd5afce-97a4-4a99-b303-3e542ef0ddba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk"
Nov 01 06:28:54.728838 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.728807    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vngm5\" (UniqueName: \"kubernetes.io/projected/0f2b2a1a-9247-47c1-bf89-65210876c41f-kube-api-access-vngm5\") pod \"payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm\" (UID: \"0f2b2a1a-9247-47c1-bf89-65210876c41f\") " pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm"
Nov 01 06:28:54.728982 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.728857    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f2b2a1a-9247-47c1-bf89-65210876c41f-plugins-config-volume\") pod \"payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm\" (UID: \"0f2b2a1a-9247-47c1-bf89-65210876c41f\") " pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm"
Nov 01 06:28:54.729554 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.729535    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f2b2a1a-9247-47c1-bf89-65210876c41f-plugins-config-volume\") pod \"payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm\" (UID: \"0f2b2a1a-9247-47c1-bf89-65210876c41f\") " pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm"
Nov 01 06:28:54.744462 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.744442    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk"
Nov 01 06:28:54.791995 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.791959    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vngm5\" (UniqueName: \"kubernetes.io/projected/0f2b2a1a-9247-47c1-bf89-65210876c41f-kube-api-access-vngm5\") pod \"payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm\" (UID: \"0f2b2a1a-9247-47c1-bf89-65210876c41f\") " pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm"
Nov 01 06:28:54.919127 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:54.919093    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm"
Nov 01 06:28:55.119783 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:55.119752    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk"]
Nov 01 06:28:55.120758 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:28:55.120729    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd5afce_97a4_4a99_b303_3e542ef0ddba.slice/crio-07b1090600579ed673070b80f63821518c98c1208d7e6efcdf149b390e9a408b WatchSource:0}: Error finding container 07b1090600579ed673070b80f63821518c98c1208d7e6efcdf149b390e9a408b: Status 404 returned error can't find the container with id 07b1090600579ed673070b80f63821518c98c1208d7e6efcdf149b390e9a408b
Nov 01 06:28:55.299101 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:55.299072    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm"]
Nov 01 06:28:55.498868 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:55.498829    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm" event={"ID":"0f2b2a1a-9247-47c1-bf89-65210876c41f","Type":"ContainerStarted","Data":"afa4a23afcb4cb57ef867d9c8995d2c0bfe82a7c2c12e8b71d1f77a56058877b"}
Nov 01 06:28:55.499062 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:55.498917    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm" event={"ID":"0f2b2a1a-9247-47c1-bf89-65210876c41f","Type":"ContainerStarted","Data":"c4bf272d7d6745b21987652d0ae855d0e8a9da5dfdbcb2025a0fe7b74454cfee"}
Nov 01 06:28:55.499062 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:55.498957    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm"
Nov 01 06:28:55.500637 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:55.500606    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk" event={"ID":"0dd5afce-97a4-4a99-b303-3e542ef0ddba","Type":"ContainerStarted","Data":"8d1dbf22633101406f7f3bf6d767081aa359032401e316f88a48e9c20a0faff1"}
Nov 01 06:28:55.500781 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:55.500658    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk" event={"ID":"0dd5afce-97a4-4a99-b303-3e542ef0ddba","Type":"ContainerStarted","Data":"07b1090600579ed673070b80f63821518c98c1208d7e6efcdf149b390e9a408b"}
Nov 01 06:28:55.500781 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:55.500685    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk"
Nov 01 06:28:55.617803 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:55.617748    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm" podStartSLOduration=1.6177292749999999 podStartE2EDuration="1.617729275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:28:55.615449601 +0000 UTC m=+2572.826273110" watchObservedRunningTime="2026-11-01 06:28:55.617729275 +0000 UTC m=+2572.828552788"
Nov 01 06:28:55.736413 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:55.736349    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk" podStartSLOduration=1.736313231 podStartE2EDuration="1.736313231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:28:55.732244909 +0000 UTC m=+2572.943068420" watchObservedRunningTime="2026-11-01 06:28:55.736313231 +0000 UTC m=+2572.947136741"
Nov 01 06:28:55.884338 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:28:55.884234    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:28:57.552744 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:28:57.552705    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:02.284842 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:02.284805    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:03.956413 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:03.956374    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:29:06.509820 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:06.509791    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm"
Nov 01 06:29:06.510260 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:06.509858    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk"
Nov 01 06:29:07.066646 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:07.066610    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:11.850387 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:11.850356    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:13.956268 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:13.956230    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:29:16.558343 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:16.558283    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:21.375028 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:21.374992    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:22.664575 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:29:22.664542    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-47cacd68a67a: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:29:22.664575 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:29:22.664567    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:29:22.665116 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:29:22.664618    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials podName:8c54c622-970a-4d9a-8bb6-06803bcf1a42 nodeName:}" failed. No retries permitted until 2026-11-01 06:31:24.664602911 +0000 UTC m=+2721.875426411 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7" (UID: "8c54c622-970a-4d9a-8bb6-06803bcf1a42") : secret "agc-cred-47cacd68a67a" not found
Nov 01 06:29:23.956180 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:23.956143    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:29:25.880002 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:29:25.879969    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:29:26.067957 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:26.067919    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:26.384429 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:26.384389    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:30.089672 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.089636    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"]
Nov 01 06:29:30.090051 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.089839    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f" podUID="10a53273-5b35-4b72-a946-7253763e3168" containerName="maas-api" containerID="cri-o://b7590f28b2c9364a35d779549ab5e296e9b2b047aa6ebc55cacbdb27403babee" gracePeriod=30
Nov 01 06:29:30.197079 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.197042    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:30.382453 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.382421    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"
Nov 01 06:29:30.458684 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.458651    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a53273-5b35-4b72-a946-7253763e3168-kube-api-access-zxwbp\" (UniqueName: \"kubernetes.io/projected/10a53273-5b35-4b72-a946-7253763e3168-kube-api-access-zxwbp\") pod \"10a53273-5b35-4b72-a946-7253763e3168\" (UID: \"10a53273-5b35-4b72-a946-7253763e3168\") "
Nov 01 06:29:30.458843 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.458748    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10a53273-5b35-4b72-a946-7253763e3168-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10a53273-5b35-4b72-a946-7253763e3168-maas-api-tls\") pod \"10a53273-5b35-4b72-a946-7253763e3168\" (UID: \"10a53273-5b35-4b72-a946-7253763e3168\") "
Nov 01 06:29:30.458892 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.458877    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10a53273-5b35-4b72-a946-7253763e3168-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a53273-5b35-4b72-a946-7253763e3168-metrics-certs\") pod \"10a53273-5b35-4b72-a946-7253763e3168\" (UID: \"10a53273-5b35-4b72-a946-7253763e3168\") "
Nov 01 06:29:30.458963 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.458947    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10a53273-5b35-4b72-a946-7253763e3168-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a53273-5b35-4b72-a946-7253763e3168-openshift-service-ca\") pod \"10a53273-5b35-4b72-a946-7253763e3168\" (UID: \"10a53273-5b35-4b72-a946-7253763e3168\") "
Nov 01 06:29:30.459380 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.459355    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a53273-5b35-4b72-a946-7253763e3168-openshift-service-ca" pod "10a53273-5b35-4b72-a946-7253763e3168" (UID: "10a53273-5b35-4b72-a946-7253763e3168"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:29:30.459575 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.459557    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a53273-5b35-4b72-a946-7253763e3168-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:29:30.461100 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.461057    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a53273-5b35-4b72-a946-7253763e3168-metrics-certs" pod "10a53273-5b35-4b72-a946-7253763e3168" (UID: "10a53273-5b35-4b72-a946-7253763e3168"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:29:30.461364 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.461340    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a53273-5b35-4b72-a946-7253763e3168-maas-api-tls" pod "10a53273-5b35-4b72-a946-7253763e3168" (UID: "10a53273-5b35-4b72-a946-7253763e3168"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:29:30.461766 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.461736    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a53273-5b35-4b72-a946-7253763e3168-kube-api-access-zxwbp" pod "10a53273-5b35-4b72-a946-7253763e3168" (UID: "10a53273-5b35-4b72-a946-7253763e3168"). InnerVolumeSpecName "kube-api-access-zxwbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:29:30.560289 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.560259    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxwbp\" (UniqueName: \"kubernetes.io/projected/10a53273-5b35-4b72-a946-7253763e3168-kube-api-access-zxwbp\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:29:30.560289 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.560289    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10a53273-5b35-4b72-a946-7253763e3168-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:29:30.560289 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.560299    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a53273-5b35-4b72-a946-7253763e3168-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:29:30.727084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.726999    5619 generic.go:356] "Generic (PLEG): container finished" podID="10a53273-5b35-4b72-a946-7253763e3168" containerID="b7590f28b2c9364a35d779549ab5e296e9b2b047aa6ebc55cacbdb27403babee" exitCode=0
Nov 01 06:29:30.727084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.727058    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"
Nov 01 06:29:30.727335 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.727060    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f" event={"ID":"10a53273-5b35-4b72-a946-7253763e3168","Type":"ContainerDied","Data":"b7590f28b2c9364a35d779549ab5e296e9b2b047aa6ebc55cacbdb27403babee"}
Nov 01 06:29:30.727335 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.727211    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f" event={"ID":"10a53273-5b35-4b72-a946-7253763e3168","Type":"ContainerDied","Data":"df8dd801ef6b436f732db4c527d3c265a77810f1383155906173e20bd5bf7f30"}
Nov 01 06:29:30.727335 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.727244    5619 scope.go:122] "RemoveContainer" containerID="b7590f28b2c9364a35d779549ab5e296e9b2b047aa6ebc55cacbdb27403babee"
Nov 01 06:29:30.739057 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.739041    5619 scope.go:122] "RemoveContainer" containerID="b7590f28b2c9364a35d779549ab5e296e9b2b047aa6ebc55cacbdb27403babee"
Nov 01 06:29:30.739316 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:29:30.739298    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7590f28b2c9364a35d779549ab5e296e9b2b047aa6ebc55cacbdb27403babee\": container with ID starting with b7590f28b2c9364a35d779549ab5e296e9b2b047aa6ebc55cacbdb27403babee not found: ID does not exist" containerID="b7590f28b2c9364a35d779549ab5e296e9b2b047aa6ebc55cacbdb27403babee"
Nov 01 06:29:30.739388 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.739338    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7590f28b2c9364a35d779549ab5e296e9b2b047aa6ebc55cacbdb27403babee"} err="failed to get container status \"b7590f28b2c9364a35d779549ab5e296e9b2b047aa6ebc55cacbdb27403babee\": rpc error: code = NotFound desc = could not find container \"b7590f28b2c9364a35d779549ab5e296e9b2b047aa6ebc55cacbdb27403babee\": container with ID starting with b7590f28b2c9364a35d779549ab5e296e9b2b047aa6ebc55cacbdb27403babee not found: ID does not exist"
Nov 01 06:29:30.894202 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.894170    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"]
Nov 01 06:29:30.945117 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.945088    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce346503-69ccffd4c8-cxt9f"]
Nov 01 06:29:30.962022 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:30.961992    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10a53273-5b35-4b72-a946-7253763e3168" path="/var/lib/kubelet/pods/10a53273-5b35-4b72-a946-7253763e3168/volumes"
Nov 01 06:29:31.099347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:31.099307    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:31.614336 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:31.614298    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"]
Nov 01 06:29:31.614965 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:31.614948    5619 cpu_manager.go:414] "containerMap: removing container" podUID="10a53273-5b35-4b72-a946-7253763e3168" containerName="maas-api"
Nov 01 06:29:31.615021 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:31.614968    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10a53273-5b35-4b72-a946-7253763e3168" containerName="maas-api"
Nov 01 06:29:31.615085 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:31.615075    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="10a53273-5b35-4b72-a946-7253763e3168" containerName="maas-api"
Nov 01 06:29:31.619814 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:31.619795    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"
Nov 01 06:29:31.678088 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:31.678053    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"]
Nov 01 06:29:31.679500 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:31.679481    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-d7d444-serving-cert\""
Nov 01 06:29:31.685486 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:31.685454    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:31.773870 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:31.773837    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g\" (UID: \"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"
Nov 01 06:29:31.774028 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:31.773900    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g\" (UID: \"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"
Nov 01 06:29:31.774104 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:31.774079    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pln2k\" (UniqueName: \"kubernetes.io/projected/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-kube-api-access-pln2k\") pod \"maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g\" (UID: \"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"
Nov 01 06:29:31.774200 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:31.774183    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g\" (UID: \"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"
Nov 01 06:29:31.876343 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:31.876243    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g\" (UID: \"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"
Nov 01 06:29:31.876479 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:31.876409    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pln2k\" (UniqueName: \"kubernetes.io/projected/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-kube-api-access-pln2k\") pod \"maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g\" (UID: \"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"
Nov 01 06:29:31.876479 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:31.876469    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g\" (UID: \"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"
Nov 01 06:29:31.876610 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:31.876596    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g\" (UID: \"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"
Nov 01 06:29:31.876799 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:29:31.876774    5619 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-serving-cert: secret "maas-api-e2e-shared-a-w1-d7d444-serving-cert" not found
Nov 01 06:29:31.876957 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:29:31.876863    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-maas-api-tls podName:1b1d6695-9e95-46fe-b1c4-f6f4b7573f67 nodeName:}" failed. No retries permitted until 2026-11-01 06:29:32.37684101 +0000 UTC m=+2609.587664516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-maas-api-tls") pod "maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g" (UID: "1b1d6695-9e95-46fe-b1c4-f6f4b7573f67") : secret "maas-api-e2e-shared-a-w1-d7d444-serving-cert" not found
Nov 01 06:29:31.877288 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:31.877262    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g\" (UID: \"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"
Nov 01 06:29:31.879299 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:31.879277    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g\" (UID: \"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"
Nov 01 06:29:31.879456 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:31.879429    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:31.939210 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:31.939167    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pln2k\" (UniqueName: \"kubernetes.io/projected/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-kube-api-access-pln2k\") pod \"maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g\" (UID: \"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"
Nov 01 06:29:32.019139 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:32.019107    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:32.163098 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:32.162995    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:32.383371 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:32.383338    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g\" (UID: \"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"
Nov 01 06:29:32.386138 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:32.386099    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g\" (UID: \"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"
Nov 01 06:29:32.537339 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:32.537247    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"
Nov 01 06:29:32.777523 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:32.777482    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:32.921094 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:32.921055    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"]
Nov 01 06:29:32.930091 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:32.930068    5619 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 06:29:33.390150 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:33.390106    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:33.749866 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:33.749791    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g" event={"ID":"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67","Type":"ContainerStarted","Data":"63a8bcb9afd314b131b33484a356494ef072264ee5e65bc8a4f6f036c334d03a"}
Nov 01 06:29:33.983385 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:33.968013    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:29:34.019620 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:34.019371    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:34.247103 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:34.247071    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:34.758762 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:34.758727    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g" event={"ID":"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67","Type":"ContainerStarted","Data":"4ca161de389a15eb2f6f89991120695d8a69a0cea0e6ab5eaba172951b0e329e"}
Nov 01 06:29:34.759242 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:34.758919    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"
Nov 01 06:29:34.878911 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:34.878854    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g" podStartSLOduration=3.0580845 podStartE2EDuration="3.878839599s" totalImagesPullingTime="820.755099ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:29:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:29:34.87721664 +0000 UTC m=+2612.088040150" watchObservedRunningTime="2026-11-01 06:29:34.878839599 +0000 UTC m=+2612.089663107"
Nov 01 06:29:35.955745 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:35.955705    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:29:37.199754 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:37.199715    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:37.450649 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:37.450564    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:37.664619 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:37.664583    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:37.813143 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:37.813049    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:40.770868 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:40.770835    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"
Nov 01 06:29:42.256122 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:42.256077    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:42.405111 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:42.405071    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:42.581719 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:42.581632    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:42.821814 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:42.821780    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv"]
Nov 01 06:29:42.825992 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:42.825972    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv"
Nov 01 06:29:42.880883 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:42.880858    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-0aed62e2\""
Nov 01 06:29:42.950948 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:42.936565    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-0aed62e2-dockercfg-qjmzt\""
Nov 01 06:29:42.969990 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:42.969951    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv"]
Nov 01 06:29:43.019816 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:43.019779    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbtmg\" (UniqueName: \"kubernetes.io/projected/a43f8f69-2a4e-40bc-8f81-154aef4e9312-kube-api-access-bbtmg\") pod \"payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv\" (UID: \"a43f8f69-2a4e-40bc-8f81-154aef4e9312\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv"
Nov 01 06:29:43.020007 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:43.019924    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43f8f69-2a4e-40bc-8f81-154aef4e9312-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv\" (UID: \"a43f8f69-2a4e-40bc-8f81-154aef4e9312\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv"
Nov 01 06:29:43.121442 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:43.121403    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbtmg\" (UniqueName: \"kubernetes.io/projected/a43f8f69-2a4e-40bc-8f81-154aef4e9312-kube-api-access-bbtmg\") pod \"payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv\" (UID: \"a43f8f69-2a4e-40bc-8f81-154aef4e9312\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv"
Nov 01 06:29:43.121620 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:43.121507    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43f8f69-2a4e-40bc-8f81-154aef4e9312-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv\" (UID: \"a43f8f69-2a4e-40bc-8f81-154aef4e9312\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv"
Nov 01 06:29:43.122207 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:43.122183    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43f8f69-2a4e-40bc-8f81-154aef4e9312-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv\" (UID: \"a43f8f69-2a4e-40bc-8f81-154aef4e9312\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv"
Nov 01 06:29:43.184607 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:43.184577    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbtmg\" (UniqueName: \"kubernetes.io/projected/a43f8f69-2a4e-40bc-8f81-154aef4e9312-kube-api-access-bbtmg\") pod \"payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv\" (UID: \"a43f8f69-2a4e-40bc-8f81-154aef4e9312\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv"
Nov 01 06:29:43.439708 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:43.439612    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv"
Nov 01 06:29:43.823634 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:29:43.823602    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43f8f69_2a4e_40bc_8f81_154aef4e9312.slice/crio-7b65d022589acb3e73ee7fdbb73706d626a9d71ca303dffcfd0cbcda864c40ea WatchSource:0}: Error finding container 7b65d022589acb3e73ee7fdbb73706d626a9d71ca303dffcfd0cbcda864c40ea: Status 404 returned error can't find the container with id 7b65d022589acb3e73ee7fdbb73706d626a9d71ca303dffcfd0cbcda864c40ea
Nov 01 06:29:43.823789 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:43.823764    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv"]
Nov 01 06:29:44.824110 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:44.824066    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv" event={"ID":"a43f8f69-2a4e-40bc-8f81-154aef4e9312","Type":"ContainerStarted","Data":"e9ec7f424bb7cbf1f26df7e012c9f8f4d3944d31834b75b2116d68b5ca8f201a"}
Nov 01 06:29:44.824603 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:44.824135    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv" event={"ID":"a43f8f69-2a4e-40bc-8f81-154aef4e9312","Type":"ContainerStarted","Data":"7b65d022589acb3e73ee7fdbb73706d626a9d71ca303dffcfd0cbcda864c40ea"}
Nov 01 06:29:44.824603 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:44.824163    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv"
Nov 01 06:29:44.943282 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:44.943233    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv" podStartSLOduration=2.943222011 podStartE2EDuration="2.943222011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:29:44.941001802 +0000 UTC m=+2622.151825310" watchObservedRunningTime="2026-11-01 06:29:44.943222011 +0000 UTC m=+2622.154045519"
Nov 01 06:29:45.956626 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:45.956589    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:29:47.374384 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:47.374344    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:52.022775 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:52.022736    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:52.216652 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:52.216609    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:54.605468 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:54.605427    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:29:55.833682 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:55.833601    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv"
Nov 01 06:29:55.880513 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:29:55.880477    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:29:55.955890 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:55.955848    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:29:56.906028 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:29:56.905993    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:01.655278 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:01.655230    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:04.916477 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:04.916439    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:05.117638 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:05.117605    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:05.955637 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:05.955596    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:30:06.391193 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:06.391153    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:09.541134 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:09.541092    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:11.147011 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:11.146978    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:15.855924 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:15.855889    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:15.964701 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:15.964660    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:30:18.873047 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:18.873010    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:20.627793 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:20.627749    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:25.005700 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:25.005667    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:25.374527 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:25.374493    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:25.881779 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:30:25.881745    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:30:25.955956 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:25.955926    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:30:26.008594 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:26.008559    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5"]
Nov 01 06:30:26.114843 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:26.114810    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g"]
Nov 01 06:30:26.374230 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:26.374198    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:26.523240 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:26.523205    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:26.750317 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:26.750229    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:30.356945 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:30.356903    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:30.819231 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:30.819145    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:31.009441 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:31.009399    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5" podUID="4c806c76-debc-4f3d-88dc-da5ec7b978b0" containerName="payload-processing" containerID="cri-o://f8c3e5407926d3505aa74ea5e8446995b119ef69de633de0d81d9be72e21eb18" gracePeriod=25
Nov 01 06:30:31.115490 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:31.115452    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g" podUID="5cbf3386-2dbf-4f96-8f0a-a7d90fe862df" containerName="payload-pre-processing" containerID="cri-o://dfdafaabd04108ee8a30e74dc5ddea3d4aa906fcef6c99bc0cbe9218f6bf8f95" gracePeriod=25
Nov 01 06:30:31.327515 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:31.327489    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5"
Nov 01 06:30:31.418451 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:31.418425    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g"
Nov 01 06:30:31.423688 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:31.423668    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c806c76-debc-4f3d-88dc-da5ec7b978b0-kube-api-access-wjt8f\" (UniqueName: \"kubernetes.io/projected/4c806c76-debc-4f3d-88dc-da5ec7b978b0-kube-api-access-wjt8f\") pod \"4c806c76-debc-4f3d-88dc-da5ec7b978b0\" (UID: \"4c806c76-debc-4f3d-88dc-da5ec7b978b0\") "
Nov 01 06:30:31.423761 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:31.423701    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c806c76-debc-4f3d-88dc-da5ec7b978b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c806c76-debc-4f3d-88dc-da5ec7b978b0-plugins-config-volume\") pod \"4c806c76-debc-4f3d-88dc-da5ec7b978b0\" (UID: \"4c806c76-debc-4f3d-88dc-da5ec7b978b0\") "
Nov 01 06:30:31.424073 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:31.424053    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c806c76-debc-4f3d-88dc-da5ec7b978b0-plugins-config-volume" pod "4c806c76-debc-4f3d-88dc-da5ec7b978b0" (UID: "4c806c76-debc-4f3d-88dc-da5ec7b978b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:30:31.425791 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:31.425767    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c806c76-debc-4f3d-88dc-da5ec7b978b0-kube-api-access-wjt8f" pod "4c806c76-debc-4f3d-88dc-da5ec7b978b0" (UID: "4c806c76-debc-4f3d-88dc-da5ec7b978b0"). InnerVolumeSpecName "kube-api-access-wjt8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:30:31.473930 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:31.473896    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:31.525009 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:31.524982    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cbf3386-2dbf-4f96-8f0a-a7d90fe862df-kube-api-access-nvfnq\" (UniqueName: \"kubernetes.io/projected/5cbf3386-2dbf-4f96-8f0a-a7d90fe862df-kube-api-access-nvfnq\") pod \"5cbf3386-2dbf-4f96-8f0a-a7d90fe862df\" (UID: \"5cbf3386-2dbf-4f96-8f0a-a7d90fe862df\") "
Nov 01 06:30:31.525164 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:31.525080    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cbf3386-2dbf-4f96-8f0a-a7d90fe862df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cbf3386-2dbf-4f96-8f0a-a7d90fe862df-plugins-config-volume\") pod \"5cbf3386-2dbf-4f96-8f0a-a7d90fe862df\" (UID: \"5cbf3386-2dbf-4f96-8f0a-a7d90fe862df\") "
Nov 01 06:30:31.525497 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:31.525466    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cbf3386-2dbf-4f96-8f0a-a7d90fe862df-plugins-config-volume" pod "5cbf3386-2dbf-4f96-8f0a-a7d90fe862df" (UID: "5cbf3386-2dbf-4f96-8f0a-a7d90fe862df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:30:31.525621 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:31.525513    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjt8f\" (UniqueName: \"kubernetes.io/projected/4c806c76-debc-4f3d-88dc-da5ec7b978b0-kube-api-access-wjt8f\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:31.525621 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:31.525529    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c806c76-debc-4f3d-88dc-da5ec7b978b0-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:31.527092 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:31.527073    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cbf3386-2dbf-4f96-8f0a-a7d90fe862df-kube-api-access-nvfnq" pod "5cbf3386-2dbf-4f96-8f0a-a7d90fe862df" (UID: "5cbf3386-2dbf-4f96-8f0a-a7d90fe862df"). InnerVolumeSpecName "kube-api-access-nvfnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:30:31.626563 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:31.626534    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvfnq\" (UniqueName: \"kubernetes.io/projected/5cbf3386-2dbf-4f96-8f0a-a7d90fe862df-kube-api-access-nvfnq\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:31.626563 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:31.626559    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cbf3386-2dbf-4f96-8f0a-a7d90fe862df-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:32.110807 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:32.110769    5619 generic.go:356] "Generic (PLEG): container finished" podID="5cbf3386-2dbf-4f96-8f0a-a7d90fe862df" containerID="dfdafaabd04108ee8a30e74dc5ddea3d4aa906fcef6c99bc0cbe9218f6bf8f95" exitCode=0
Nov 01 06:30:32.111003 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:32.110827    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g"
Nov 01 06:30:32.111003 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:32.110851    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g" event={"ID":"5cbf3386-2dbf-4f96-8f0a-a7d90fe862df","Type":"ContainerDied","Data":"dfdafaabd04108ee8a30e74dc5ddea3d4aa906fcef6c99bc0cbe9218f6bf8f95"}
Nov 01 06:30:32.111003 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:32.110907    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g" event={"ID":"5cbf3386-2dbf-4f96-8f0a-a7d90fe862df","Type":"ContainerDied","Data":"44f7f987f7f409d6b1decfad9bd2716e39bcdd390fa26c2331c95daff70462f2"}
Nov 01 06:30:32.111003 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:32.110933    5619 scope.go:122] "RemoveContainer" containerID="dfdafaabd04108ee8a30e74dc5ddea3d4aa906fcef6c99bc0cbe9218f6bf8f95"
Nov 01 06:30:32.112506 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:32.112484    5619 generic.go:356] "Generic (PLEG): container finished" podID="4c806c76-debc-4f3d-88dc-da5ec7b978b0" containerID="f8c3e5407926d3505aa74ea5e8446995b119ef69de633de0d81d9be72e21eb18" exitCode=0
Nov 01 06:30:32.112636 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:32.112533    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5"
Nov 01 06:30:32.112636 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:32.112600    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5" event={"ID":"4c806c76-debc-4f3d-88dc-da5ec7b978b0","Type":"ContainerDied","Data":"f8c3e5407926d3505aa74ea5e8446995b119ef69de633de0d81d9be72e21eb18"}
Nov 01 06:30:32.112722 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:32.112637    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5" event={"ID":"4c806c76-debc-4f3d-88dc-da5ec7b978b0","Type":"ContainerDied","Data":"0933196ab9c39af7576a6280548878c2151160c265932a02413493ddec58ed73"}
Nov 01 06:30:32.122374 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:32.122354    5619 scope.go:122] "RemoveContainer" containerID="dfdafaabd04108ee8a30e74dc5ddea3d4aa906fcef6c99bc0cbe9218f6bf8f95"
Nov 01 06:30:32.122668 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:30:32.122641    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfdafaabd04108ee8a30e74dc5ddea3d4aa906fcef6c99bc0cbe9218f6bf8f95\": container with ID starting with dfdafaabd04108ee8a30e74dc5ddea3d4aa906fcef6c99bc0cbe9218f6bf8f95 not found: ID does not exist" containerID="dfdafaabd04108ee8a30e74dc5ddea3d4aa906fcef6c99bc0cbe9218f6bf8f95"
Nov 01 06:30:32.122758 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:32.122674    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfdafaabd04108ee8a30e74dc5ddea3d4aa906fcef6c99bc0cbe9218f6bf8f95"} err="failed to get container status \"dfdafaabd04108ee8a30e74dc5ddea3d4aa906fcef6c99bc0cbe9218f6bf8f95\": rpc error: code = NotFound desc = could not find container \"dfdafaabd04108ee8a30e74dc5ddea3d4aa906fcef6c99bc0cbe9218f6bf8f95\": container with ID starting with dfdafaabd04108ee8a30e74dc5ddea3d4aa906fcef6c99bc0cbe9218f6bf8f95 not found: ID does not exist"
Nov 01 06:30:32.122758 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:32.122694    5619 scope.go:122] "RemoveContainer" containerID="f8c3e5407926d3505aa74ea5e8446995b119ef69de633de0d81d9be72e21eb18"
Nov 01 06:30:32.132410 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:32.132393    5619 scope.go:122] "RemoveContainer" containerID="f8c3e5407926d3505aa74ea5e8446995b119ef69de633de0d81d9be72e21eb18"
Nov 01 06:30:32.132683 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:30:32.132663    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c3e5407926d3505aa74ea5e8446995b119ef69de633de0d81d9be72e21eb18\": container with ID starting with f8c3e5407926d3505aa74ea5e8446995b119ef69de633de0d81d9be72e21eb18 not found: ID does not exist" containerID="f8c3e5407926d3505aa74ea5e8446995b119ef69de633de0d81d9be72e21eb18"
Nov 01 06:30:32.132737 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:32.132691    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c3e5407926d3505aa74ea5e8446995b119ef69de633de0d81d9be72e21eb18"} err="failed to get container status \"f8c3e5407926d3505aa74ea5e8446995b119ef69de633de0d81d9be72e21eb18\": rpc error: code = NotFound desc = could not find container \"f8c3e5407926d3505aa74ea5e8446995b119ef69de633de0d81d9be72e21eb18\": container with ID starting with f8c3e5407926d3505aa74ea5e8446995b119ef69de633de0d81d9be72e21eb18 not found: ID does not exist"
Nov 01 06:30:32.279352 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:32.279266    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5"]
Nov 01 06:30:32.330688 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:32.330658    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f13905-77dc5c4667-mq4p5"]
Nov 01 06:30:32.403535 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:32.403500    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g"]
Nov 01 06:30:32.405523 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:32.405496    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f13905-5fdd6f5c5-cqg5g"]
Nov 01 06:30:32.962962 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:32.962929    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c806c76-debc-4f3d-88dc-da5ec7b978b0" path="/var/lib/kubelet/pods/4c806c76-debc-4f3d-88dc-da5ec7b978b0/volumes"
Nov 01 06:30:32.963360 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:32.963304    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cbf3386-2dbf-4f96-8f0a-a7d90fe862df" path="/var/lib/kubelet/pods/5cbf3386-2dbf-4f96-8f0a-a7d90fe862df/volumes"
Nov 01 06:30:35.077815 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:35.077783    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:35.955888 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:35.955851    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:30:36.206600 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:36.206518    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:36.401482 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:36.401436    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:39.873231 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:39.873193    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:40.478200 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.478163    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"]
Nov 01 06:30:40.479425 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.479394    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:30:40.528918 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.528890    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"]
Nov 01 06:30:40.529738 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.529714    5619 cpu_manager.go:414] "containerMap: removing container" podUID="4c806c76-debc-4f3d-88dc-da5ec7b978b0" containerName="payload-processing"
Nov 01 06:30:40.529738 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.529736    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c806c76-debc-4f3d-88dc-da5ec7b978b0" containerName="payload-processing"
Nov 01 06:30:40.529916 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.529778    5619 cpu_manager.go:414] "containerMap: removing container" podUID="5cbf3386-2dbf-4f96-8f0a-a7d90fe862df" containerName="payload-pre-processing"
Nov 01 06:30:40.529916 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.529785    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cbf3386-2dbf-4f96-8f0a-a7d90fe862df" containerName="payload-pre-processing"
Nov 01 06:30:40.529916 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.529887    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c806c76-debc-4f3d-88dc-da5ec7b978b0" containerName="payload-processing"
Nov 01 06:30:40.529916 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.529900    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cbf3386-2dbf-4f96-8f0a-a7d90fe862df" containerName="payload-pre-processing"
Nov 01 06:30:40.534401 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.534380    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"
Nov 01 06:30:40.588131 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.588103    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"agc-cred-089523cf8a24\""
Nov 01 06:30:40.591648 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.591620    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"]
Nov 01 06:30:40.636012 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.635977    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf\" (UID: \"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"
Nov 01 06:30:40.636390 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.636361    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf\" (UID: \"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"
Nov 01 06:30:40.636524 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.636410    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf\" (UID: \"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"
Nov 01 06:30:40.636524 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.636465    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzvbb\" (UniqueName: \"kubernetes.io/projected/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-kube-api-access-rzvbb\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf\" (UID: \"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"
Nov 01 06:30:40.737809 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.737724    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf\" (UID: \"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"
Nov 01 06:30:40.737958 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.737911    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf\" (UID: \"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"
Nov 01 06:30:40.737958 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.737936    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf\" (UID: \"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"
Nov 01 06:30:40.738382 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.737954    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzvbb\" (UniqueName: \"kubernetes.io/projected/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-kube-api-access-rzvbb\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf\" (UID: \"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"
Nov 01 06:30:40.738778 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.738755    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf\" (UID: \"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"
Nov 01 06:30:40.738851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.738837    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf\" (UID: \"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"
Nov 01 06:30:40.740311 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.740285    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf\" (UID: \"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"
Nov 01 06:30:40.800007 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.799982    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzvbb\" (UniqueName: \"kubernetes.io/projected/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-kube-api-access-rzvbb\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf\" (UID: \"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"
Nov 01 06:30:40.848878 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:40.848852    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"
Nov 01 06:30:41.223801 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:41.223768    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"]
Nov 01 06:30:41.225317 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:30:41.225291    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd031b4_43b6_4cc6_9991_9f1c92f6fb7a.slice/crio-bc62c1e205e5ebf7963c7e13ffce609f30f16de03daa8e410492dd4ffcd537c6 WatchSource:0}: Error finding container bc62c1e205e5ebf7963c7e13ffce609f30f16de03daa8e410492dd4ffcd537c6: Status 404 returned error can't find the container with id bc62c1e205e5ebf7963c7e13ffce609f30f16de03daa8e410492dd4ffcd537c6
Nov 01 06:30:42.190128 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:42.190084    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf" event={"ID":"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a","Type":"ContainerStarted","Data":"ec1a9f518ffab195864711366b1319fcb44e39bda5ee45caed5c4298cd013cde"}
Nov 01 06:30:42.190314 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:42.190188    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"
Nov 01 06:30:42.190314 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:42.190212    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf" event={"ID":"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a","Type":"ContainerStarted","Data":"bc62c1e205e5ebf7963c7e13ffce609f30f16de03daa8e410492dd4ffcd537c6"}
Nov 01 06:30:42.308766 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:42.308716    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf" podStartSLOduration=2.308699346 podStartE2EDuration="2.308699346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:42.307783951 +0000 UTC m=+2679.518607460" watchObservedRunningTime="2026-11-01 06:30:42.308699346 +0000 UTC m=+2679.519522854"
Nov 01 06:30:44.324211 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:44.324181    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"]
Nov 01 06:30:44.328456 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:44.328436    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"
Nov 01 06:30:44.382371 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:44.382308    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcvms\" (UniqueName: \"kubernetes.io/projected/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-kube-api-access-pcvms\") pod \"maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7\" (UID: \"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"
Nov 01 06:30:44.382511 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:44.382404    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-metrics-certs\") pod \"maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7\" (UID: \"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"
Nov 01 06:30:44.382511 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:44.382471    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-maas-api-tls\") pod \"maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7\" (UID: \"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"
Nov 01 06:30:44.382615 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:44.382599    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-openshift-service-ca\") pod \"maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7\" (UID: \"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"
Nov 01 06:30:44.384267 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:44.384240    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-0aed62e2-serving-cert\""
Nov 01 06:30:44.437809 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:44.437778    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"]
Nov 01 06:30:44.483259 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:44.483228    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-openshift-service-ca\") pod \"maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7\" (UID: \"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"
Nov 01 06:30:44.483429 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:44.483363    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcvms\" (UniqueName: \"kubernetes.io/projected/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-kube-api-access-pcvms\") pod \"maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7\" (UID: \"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"
Nov 01 06:30:44.483429 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:44.483404    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-metrics-certs\") pod \"maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7\" (UID: \"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"
Nov 01 06:30:44.483516 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:44.483440    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-maas-api-tls\") pod \"maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7\" (UID: \"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"
Nov 01 06:30:44.483636 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:30:44.483611    5619 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-serving-cert: secret "maas-api-e2e-derive-0aed62e2-serving-cert" not found
Nov 01 06:30:44.483698 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:30:44.483686    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-maas-api-tls podName:5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8 nodeName:}" failed. No retries permitted until 2026-11-01 06:30:44.983666213 +0000 UTC m=+2682.194489704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-maas-api-tls") pod "maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7" (UID: "5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8") : secret "maas-api-e2e-derive-0aed62e2-serving-cert" not found
Nov 01 06:30:44.483977 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:44.483959    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-openshift-service-ca\") pod \"maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7\" (UID: \"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"
Nov 01 06:30:44.486059 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:44.486035    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-metrics-certs\") pod \"maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7\" (UID: \"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"
Nov 01 06:30:44.519222 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:44.519197    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:44.545518 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:44.545495    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcvms\" (UniqueName: \"kubernetes.io/projected/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-kube-api-access-pcvms\") pod \"maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7\" (UID: \"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"
Nov 01 06:30:44.678268 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:44.678232    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:44.826889 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:44.826858    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:44.990031 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:44.989952    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-maas-api-tls\") pod \"maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7\" (UID: \"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"
Nov 01 06:30:44.992561 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:44.992534    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-maas-api-tls\") pod \"maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7\" (UID: \"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"
Nov 01 06:30:45.243576 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:45.243495    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"
Nov 01 06:30:45.478769 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:45.478726    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing" containerID="cri-o://2e9c8e12e0b838630105ee7e956255f27ffc93c601e4bb72b66b60c963db38bf" gracePeriod=25
Nov 01 06:30:45.622482 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:45.622455    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"]
Nov 01 06:30:45.626938 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:30:45.626907    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8834a9_3d3d_41da_bd8e_2b8cb85c74d8.slice/crio-20a5199ae4cdb859d392e67239b2bcc51aa222f1100a96ee7d10d6f5fffff011 WatchSource:0}: Error finding container 20a5199ae4cdb859d392e67239b2bcc51aa222f1100a96ee7d10d6f5fffff011: Status 404 returned error can't find the container with id 20a5199ae4cdb859d392e67239b2bcc51aa222f1100a96ee7d10d6f5fffff011
Nov 01 06:30:45.744421 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:45.744393    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"
Nov 01 06:30:45.799162 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:45.799090    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b07ef354-09ac-41d1-af4d-fd904e5fef4f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b07ef354-09ac-41d1-af4d-fd904e5fef4f-routing-overlay-volume\") pod \"b07ef354-09ac-41d1-af4d-fd904e5fef4f\" (UID: \"b07ef354-09ac-41d1-af4d-fd904e5fef4f\") "
Nov 01 06:30:45.799162 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:45.799127    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b07ef354-09ac-41d1-af4d-fd904e5fef4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07ef354-09ac-41d1-af4d-fd904e5fef4f-plugins-config-volume\") pod \"b07ef354-09ac-41d1-af4d-fd904e5fef4f\" (UID: \"b07ef354-09ac-41d1-af4d-fd904e5fef4f\") "
Nov 01 06:30:45.799392 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:45.799180    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b07ef354-09ac-41d1-af4d-fd904e5fef4f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b07ef354-09ac-41d1-af4d-fd904e5fef4f-provider-credentials\") pod \"b07ef354-09ac-41d1-af4d-fd904e5fef4f\" (UID: \"b07ef354-09ac-41d1-af4d-fd904e5fef4f\") "
Nov 01 06:30:45.799392 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:45.799271    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b07ef354-09ac-41d1-af4d-fd904e5fef4f-kube-api-access-57wct\" (UniqueName: \"kubernetes.io/projected/b07ef354-09ac-41d1-af4d-fd904e5fef4f-kube-api-access-57wct\") pod \"b07ef354-09ac-41d1-af4d-fd904e5fef4f\" (UID: \"b07ef354-09ac-41d1-af4d-fd904e5fef4f\") "
Nov 01 06:30:45.799572 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:45.799547    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07ef354-09ac-41d1-af4d-fd904e5fef4f-routing-overlay-volume" pod "b07ef354-09ac-41d1-af4d-fd904e5fef4f" (UID: "b07ef354-09ac-41d1-af4d-fd904e5fef4f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:30:45.799754 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:45.799723    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07ef354-09ac-41d1-af4d-fd904e5fef4f-plugins-config-volume" pod "b07ef354-09ac-41d1-af4d-fd904e5fef4f" (UID: "b07ef354-09ac-41d1-af4d-fd904e5fef4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:30:45.799806 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:45.799753    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b07ef354-09ac-41d1-af4d-fd904e5fef4f-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:45.801599 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:45.801573    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07ef354-09ac-41d1-af4d-fd904e5fef4f-kube-api-access-57wct" pod "b07ef354-09ac-41d1-af4d-fd904e5fef4f" (UID: "b07ef354-09ac-41d1-af4d-fd904e5fef4f"). InnerVolumeSpecName "kube-api-access-57wct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:30:45.801735 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:45.801715    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07ef354-09ac-41d1-af4d-fd904e5fef4f-provider-credentials" pod "b07ef354-09ac-41d1-af4d-fd904e5fef4f" (UID: "b07ef354-09ac-41d1-af4d-fd904e5fef4f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:30:45.901141 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:45.901101    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07ef354-09ac-41d1-af4d-fd904e5fef4f-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:45.901141 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:45.901144    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b07ef354-09ac-41d1-af4d-fd904e5fef4f-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:45.901352 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:45.901161    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57wct\" (UniqueName: \"kubernetes.io/projected/b07ef354-09ac-41d1-af4d-fd904e5fef4f-kube-api-access-57wct\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:46.201659 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:46.201625    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:46.215977 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:46.215937    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7" event={"ID":"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8","Type":"ContainerStarted","Data":"20a5199ae4cdb859d392e67239b2bcc51aa222f1100a96ee7d10d6f5fffff011"}
Nov 01 06:30:46.217263 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:46.217232    5619 generic.go:356] "Generic (PLEG): container finished" podID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerID="2e9c8e12e0b838630105ee7e956255f27ffc93c601e4bb72b66b60c963db38bf" exitCode=0
Nov 01 06:30:46.217414 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:46.217304    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"
Nov 01 06:30:46.217414 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:46.217338    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" event={"ID":"b07ef354-09ac-41d1-af4d-fd904e5fef4f","Type":"ContainerDied","Data":"2e9c8e12e0b838630105ee7e956255f27ffc93c601e4bb72b66b60c963db38bf"}
Nov 01 06:30:46.217414 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:46.217402    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx" event={"ID":"b07ef354-09ac-41d1-af4d-fd904e5fef4f","Type":"ContainerDied","Data":"0ee7180bbd5e32618a5dcd34208b2472edd5a02bcc65339f004fbccef1322a62"}
Nov 01 06:30:46.217605 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:46.217434    5619 scope.go:122] "RemoveContainer" containerID="2e9c8e12e0b838630105ee7e956255f27ffc93c601e4bb72b66b60c963db38bf"
Nov 01 06:30:46.228260 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:46.228240    5619 scope.go:122] "RemoveContainer" containerID="2e9c8e12e0b838630105ee7e956255f27ffc93c601e4bb72b66b60c963db38bf"
Nov 01 06:30:46.228568 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:30:46.228548    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e9c8e12e0b838630105ee7e956255f27ffc93c601e4bb72b66b60c963db38bf\": container with ID starting with 2e9c8e12e0b838630105ee7e956255f27ffc93c601e4bb72b66b60c963db38bf not found: ID does not exist" containerID="2e9c8e12e0b838630105ee7e956255f27ffc93c601e4bb72b66b60c963db38bf"
Nov 01 06:30:46.228640 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:46.228577    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9c8e12e0b838630105ee7e956255f27ffc93c601e4bb72b66b60c963db38bf"} err="failed to get container status \"2e9c8e12e0b838630105ee7e956255f27ffc93c601e4bb72b66b60c963db38bf\": rpc error: code = NotFound desc = could not find container \"2e9c8e12e0b838630105ee7e956255f27ffc93c601e4bb72b66b60c963db38bf\": container with ID starting with 2e9c8e12e0b838630105ee7e956255f27ffc93c601e4bb72b66b60c963db38bf not found: ID does not exist"
Nov 01 06:30:46.385072 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:46.385034    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"]
Nov 01 06:30:46.389648 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:46.389614    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9g7sqx"]
Nov 01 06:30:46.963764 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:46.963734    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" path="/var/lib/kubelet/pods/b07ef354-09ac-41d1-af4d-fd904e5fef4f/volumes"
Nov 01 06:30:47.230625 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:47.230523    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7" event={"ID":"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8","Type":"ContainerStarted","Data":"07faa4161b816bd276187d15f5bc62caf24c5f11d835b1b5f7db4967fd1e11fb"}
Nov 01 06:30:47.230804 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:47.230644    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"
Nov 01 06:30:47.350462 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:47.350412    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7" podStartSLOduration=2.505945797 podStartE2EDuration="3.350395771s" totalImagesPullingTime="844.449974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:47.349278777 +0000 UTC m=+2684.560102287" watchObservedRunningTime="2026-11-01 06:30:47.350395771 +0000 UTC m=+2684.561219320"
Nov 01 06:30:49.769810 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:49.769766    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:53.198910 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:53.198881    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"
Nov 01 06:30:53.243124 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:53.243097    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"
Nov 01 06:30:53.371457 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:53.371420    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"]
Nov 01 06:30:54.509710 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:54.509679    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:55.880084 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:30:55.880039    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:30:56.623801 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:56.623763    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:57.281053 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:57.280825    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:57.594049 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:57.594016    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:58.372690 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:58.372645    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7" podUID="8c54c622-970a-4d9a-8bb6-06803bcf1a42" containerName="payload-processing" containerID="cri-o://0480c7e59751150338fa79c2f31348e37203981f76e1566501bdda9e4d870b2f" gracePeriod=25
Nov 01 06:30:58.628298 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:58.628235    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"
Nov 01 06:30:58.765732 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:58.765695    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c54c622-970a-4d9a-8bb6-06803bcf1a42-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c54c622-970a-4d9a-8bb6-06803bcf1a42-routing-overlay-volume\") pod \"8c54c622-970a-4d9a-8bb6-06803bcf1a42\" (UID: \"8c54c622-970a-4d9a-8bb6-06803bcf1a42\") "
Nov 01 06:30:58.765930 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:58.765808    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials\") pod \"8c54c622-970a-4d9a-8bb6-06803bcf1a42\" (UID: \"8c54c622-970a-4d9a-8bb6-06803bcf1a42\") "
Nov 01 06:30:58.765930 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:58.765886    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c54c622-970a-4d9a-8bb6-06803bcf1a42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c54c622-970a-4d9a-8bb6-06803bcf1a42-plugins-config-volume\") pod \"8c54c622-970a-4d9a-8bb6-06803bcf1a42\" (UID: \"8c54c622-970a-4d9a-8bb6-06803bcf1a42\") "
Nov 01 06:30:58.766053 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:58.765926    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-kube-api-access-sj5hc\" (UniqueName: \"kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-kube-api-access-sj5hc\") pod \"8c54c622-970a-4d9a-8bb6-06803bcf1a42\" (UID: \"8c54c622-970a-4d9a-8bb6-06803bcf1a42\") "
Nov 01 06:30:58.766183 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:58.766151    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c54c622-970a-4d9a-8bb6-06803bcf1a42-routing-overlay-volume" pod "8c54c622-970a-4d9a-8bb6-06803bcf1a42" (UID: "8c54c622-970a-4d9a-8bb6-06803bcf1a42"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:30:58.766730 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:58.766611    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c54c622-970a-4d9a-8bb6-06803bcf1a42-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:58.766730 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:58.766697    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c54c622-970a-4d9a-8bb6-06803bcf1a42-plugins-config-volume" pod "8c54c622-970a-4d9a-8bb6-06803bcf1a42" (UID: "8c54c622-970a-4d9a-8bb6-06803bcf1a42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:30:58.769001 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:58.768975    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-kube-api-access-sj5hc" pod "8c54c622-970a-4d9a-8bb6-06803bcf1a42" (UID: "8c54c622-970a-4d9a-8bb6-06803bcf1a42"). InnerVolumeSpecName "kube-api-access-sj5hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:30:58.769122 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:58.769025    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials" pod "8c54c622-970a-4d9a-8bb6-06803bcf1a42" (UID: "8c54c622-970a-4d9a-8bb6-06803bcf1a42"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:30:58.868228 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:58.868190    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:58.868228 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:58.868224    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c54c622-970a-4d9a-8bb6-06803bcf1a42-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:58.868228 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:58.868234    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj5hc\" (UniqueName: \"kubernetes.io/projected/8c54c622-970a-4d9a-8bb6-06803bcf1a42-kube-api-access-sj5hc\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:30:59.307916 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:59.307823    5619 generic.go:356] "Generic (PLEG): container finished" podID="8c54c622-970a-4d9a-8bb6-06803bcf1a42" containerID="0480c7e59751150338fa79c2f31348e37203981f76e1566501bdda9e4d870b2f" exitCode=0
Nov 01 06:30:59.307916 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:59.307882    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"
Nov 01 06:30:59.308165 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:59.307935    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7" event={"ID":"8c54c622-970a-4d9a-8bb6-06803bcf1a42","Type":"ContainerDied","Data":"0480c7e59751150338fa79c2f31348e37203981f76e1566501bdda9e4d870b2f"}
Nov 01 06:30:59.308165 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:59.308005    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7" event={"ID":"8c54c622-970a-4d9a-8bb6-06803bcf1a42","Type":"ContainerDied","Data":"2d778be3a307f43fbf8df6ab269251242c48ebb639268d4700a245410ccca295"}
Nov 01 06:30:59.308165 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:59.308038    5619 scope.go:122] "RemoveContainer" containerID="0480c7e59751150338fa79c2f31348e37203981f76e1566501bdda9e4d870b2f"
Nov 01 06:30:59.317870 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:59.317854    5619 scope.go:122] "RemoveContainer" containerID="0480c7e59751150338fa79c2f31348e37203981f76e1566501bdda9e4d870b2f"
Nov 01 06:30:59.318133 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:30:59.318104    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0480c7e59751150338fa79c2f31348e37203981f76e1566501bdda9e4d870b2f\": container with ID starting with 0480c7e59751150338fa79c2f31348e37203981f76e1566501bdda9e4d870b2f not found: ID does not exist" containerID="0480c7e59751150338fa79c2f31348e37203981f76e1566501bdda9e4d870b2f"
Nov 01 06:30:59.318182 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:59.318141    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0480c7e59751150338fa79c2f31348e37203981f76e1566501bdda9e4d870b2f"} err="failed to get container status \"0480c7e59751150338fa79c2f31348e37203981f76e1566501bdda9e4d870b2f\": rpc error: code = NotFound desc = could not find container \"0480c7e59751150338fa79c2f31348e37203981f76e1566501bdda9e4d870b2f\": container with ID starting with 0480c7e59751150338fa79c2f31348e37203981f76e1566501bdda9e4d870b2f not found: ID does not exist"
Nov 01 06:30:59.389556 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:59.389519    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:30:59.475450 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:59.475415    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"]
Nov 01 06:30:59.526260 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:30:59.526228    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f799zp7"]
Nov 01 06:31:00.963808 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:00.963773    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c54c622-970a-4d9a-8bb6-06803bcf1a42" path="/var/lib/kubelet/pods/8c54c622-970a-4d9a-8bb6-06803bcf1a42/volumes"
Nov 01 06:31:02.037849 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:02.037814    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:02.241198 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:02.241156    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:04.559584 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:04.559549    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:05.522732 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:05.522695    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:05.669431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:05.669387    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:05.896746 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:05.896711    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g"]
Nov 01 06:31:05.897410 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:05.897391    5619 cpu_manager.go:414] "containerMap: removing container" podUID="8c54c622-970a-4d9a-8bb6-06803bcf1a42" containerName="payload-processing"
Nov 01 06:31:05.897410 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:05.897410    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c54c622-970a-4d9a-8bb6-06803bcf1a42" containerName="payload-processing"
Nov 01 06:31:05.897607 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:05.897435    5619 cpu_manager.go:414] "containerMap: removing container" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing"
Nov 01 06:31:05.897607 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:05.897443    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing"
Nov 01 06:31:05.897711 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:05.897614    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c54c622-970a-4d9a-8bb6-06803bcf1a42" containerName="payload-processing"
Nov 01 06:31:05.897711 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:05.897630    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="b07ef354-09ac-41d1-af4d-fd904e5fef4f" containerName="payload-processing"
Nov 01 06:31:05.902825 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:05.902802    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g"
Nov 01 06:31:05.964251 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:05.964214    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g"]
Nov 01 06:31:06.014344 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:06.011846    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-747b7b\""
Nov 01 06:31:06.014344 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:06.012021    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-747b7b-dockercfg-s2zfn\""
Nov 01 06:31:06.033828 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:06.033770    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c"]
Nov 01 06:31:06.063234 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:06.063187    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5xs4\" (UniqueName: \"kubernetes.io/projected/dbd66390-4712-4e35-acae-26c5b43e35ef-kube-api-access-d5xs4\") pod \"payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g\" (UID: \"dbd66390-4712-4e35-acae-26c5b43e35ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g"
Nov 01 06:31:06.063579 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:06.063293    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd66390-4712-4e35-acae-26c5b43e35ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g\" (UID: \"dbd66390-4712-4e35-acae-26c5b43e35ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g"
Nov 01 06:31:06.064415 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:06.064391    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c"
Nov 01 06:31:06.086974 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:06.086939    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c"]
Nov 01 06:31:06.164150 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:06.164068    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5xs4\" (UniqueName: \"kubernetes.io/projected/dbd66390-4712-4e35-acae-26c5b43e35ef-kube-api-access-d5xs4\") pod \"payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g\" (UID: \"dbd66390-4712-4e35-acae-26c5b43e35ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g"
Nov 01 06:31:06.164150 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:06.164110    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdjsg\" (UniqueName: \"kubernetes.io/projected/6b322967-2ec4-4195-b5dc-9524a46b5091-kube-api-access-fdjsg\") pod \"payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c\" (UID: \"6b322967-2ec4-4195-b5dc-9524a46b5091\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c"
Nov 01 06:31:06.164150 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:06.164147    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd66390-4712-4e35-acae-26c5b43e35ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g\" (UID: \"dbd66390-4712-4e35-acae-26c5b43e35ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g"
Nov 01 06:31:06.164418 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:06.164183    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b322967-2ec4-4195-b5dc-9524a46b5091-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c\" (UID: \"6b322967-2ec4-4195-b5dc-9524a46b5091\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c"
Nov 01 06:31:06.165069 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:06.165042    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd66390-4712-4e35-acae-26c5b43e35ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g\" (UID: \"dbd66390-4712-4e35-acae-26c5b43e35ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g"
Nov 01 06:31:06.226517 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:06.226483    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5xs4\" (UniqueName: \"kubernetes.io/projected/dbd66390-4712-4e35-acae-26c5b43e35ef-kube-api-access-d5xs4\") pod \"payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g\" (UID: \"dbd66390-4712-4e35-acae-26c5b43e35ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g"
Nov 01 06:31:06.265580 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:06.265546    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b322967-2ec4-4195-b5dc-9524a46b5091-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c\" (UID: \"6b322967-2ec4-4195-b5dc-9524a46b5091\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c"
Nov 01 06:31:06.265740 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:06.265706    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdjsg\" (UniqueName: \"kubernetes.io/projected/6b322967-2ec4-4195-b5dc-9524a46b5091-kube-api-access-fdjsg\") pod \"payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c\" (UID: \"6b322967-2ec4-4195-b5dc-9524a46b5091\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c"
Nov 01 06:31:06.266293 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:06.266273    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b322967-2ec4-4195-b5dc-9524a46b5091-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c\" (UID: \"6b322967-2ec4-4195-b5dc-9524a46b5091\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c"
Nov 01 06:31:06.329027 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:06.328999    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdjsg\" (UniqueName: \"kubernetes.io/projected/6b322967-2ec4-4195-b5dc-9524a46b5091-kube-api-access-fdjsg\") pod \"payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c\" (UID: \"6b322967-2ec4-4195-b5dc-9524a46b5091\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c"
Nov 01 06:31:06.384147 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:06.384120    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c"
Nov 01 06:31:06.517914 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:06.517889    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g"
Nov 01 06:31:06.767360 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:06.767338    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c"]
Nov 01 06:31:06.768725 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:31:06.768698    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b322967_2ec4_4195_b5dc_9524a46b5091.slice/crio-5e1687ec54551c9585ba996893e133516480e16452c52f8739c25b1b538857ba WatchSource:0}: Error finding container 5e1687ec54551c9585ba996893e133516480e16452c52f8739c25b1b538857ba: Status 404 returned error can't find the container with id 5e1687ec54551c9585ba996893e133516480e16452c52f8739c25b1b538857ba
Nov 01 06:31:06.900527 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:06.900496    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g"]
Nov 01 06:31:06.901982 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:31:06.901956    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd66390_4712_4e35_acae_26c5b43e35ef.slice/crio-503dd00979cf6c9b2b71ab3f5bd0c06970e75c555f959e7fc6c6cfcbb308f002 WatchSource:0}: Error finding container 503dd00979cf6c9b2b71ab3f5bd0c06970e75c555f959e7fc6c6cfcbb308f002: Status 404 returned error can't find the container with id 503dd00979cf6c9b2b71ab3f5bd0c06970e75c555f959e7fc6c6cfcbb308f002
Nov 01 06:31:07.359992 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:07.359957    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c" event={"ID":"6b322967-2ec4-4195-b5dc-9524a46b5091","Type":"ContainerStarted","Data":"71ed5d28285dd754bd1337cb2aacb71892965ce9c81dc1594207971fb52f8bb5"}
Nov 01 06:31:07.360185 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:07.360011    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c" event={"ID":"6b322967-2ec4-4195-b5dc-9524a46b5091","Type":"ContainerStarted","Data":"5e1687ec54551c9585ba996893e133516480e16452c52f8739c25b1b538857ba"}
Nov 01 06:31:07.360185 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:07.360041    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c"
Nov 01 06:31:07.361691 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:07.361665    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g" event={"ID":"dbd66390-4712-4e35-acae-26c5b43e35ef","Type":"ContainerStarted","Data":"648fb08fd78fde93986e74bdf344d27f12b86928acae46c369c7d9f4a4f86458"}
Nov 01 06:31:07.361808 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:07.361710    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g" event={"ID":"dbd66390-4712-4e35-acae-26c5b43e35ef","Type":"ContainerStarted","Data":"503dd00979cf6c9b2b71ab3f5bd0c06970e75c555f959e7fc6c6cfcbb308f002"}
Nov 01 06:31:07.361808 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:07.361751    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g"
Nov 01 06:31:07.484474 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:07.484420    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c" podStartSLOduration=2.484404211 podStartE2EDuration="2.484404211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:31:07.481461633 +0000 UTC m=+2704.692285142" watchObservedRunningTime="2026-11-01 06:31:07.484404211 +0000 UTC m=+2704.695227720"
Nov 01 06:31:07.599968 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:07.599884    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g" podStartSLOduration=2.5998649 podStartE2EDuration="2.5998649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:31:07.599759566 +0000 UTC m=+2704.810583073" watchObservedRunningTime="2026-11-01 06:31:07.5998649 +0000 UTC m=+2704.810688411"
Nov 01 06:31:09.568652 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:09.568618    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:10.823394 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:10.823291    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:14.608049 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:14.608006    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:17.359189 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:17.359152    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:17.510936 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:17.510894    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:18.372652 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:18.372619    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c"
Nov 01 06:31:18.373101 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:18.372692    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g"
Nov 01 06:31:19.459479 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:19.459440    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:23.491491 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:23.491455    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"]
Nov 01 06:31:23.491959 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:23.491704    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt" podUID="1312ccd3-4c59-4cd1-b087-93c9aabcd152" containerName="maas-api" containerID="cri-o://3fd48e9d111882b79078c9558af5a0a2b54fa5423e73f8561d01cce02754e8c3" gracePeriod=30
Nov 01 06:31:23.595818 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:23.595761    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:23.762652 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:23.762578    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"
Nov 01 06:31:23.896238 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:23.896194    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1312ccd3-4c59-4cd1-b087-93c9aabcd152-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1312ccd3-4c59-4cd1-b087-93c9aabcd152-openshift-service-ca\") pod \"1312ccd3-4c59-4cd1-b087-93c9aabcd152\" (UID: \"1312ccd3-4c59-4cd1-b087-93c9aabcd152\") "
Nov 01 06:31:23.896514 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:23.896297    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1312ccd3-4c59-4cd1-b087-93c9aabcd152-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1312ccd3-4c59-4cd1-b087-93c9aabcd152-maas-api-tls\") pod \"1312ccd3-4c59-4cd1-b087-93c9aabcd152\" (UID: \"1312ccd3-4c59-4cd1-b087-93c9aabcd152\") "
Nov 01 06:31:23.896514 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:23.896350    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1312ccd3-4c59-4cd1-b087-93c9aabcd152-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1312ccd3-4c59-4cd1-b087-93c9aabcd152-metrics-certs\") pod \"1312ccd3-4c59-4cd1-b087-93c9aabcd152\" (UID: \"1312ccd3-4c59-4cd1-b087-93c9aabcd152\") "
Nov 01 06:31:23.896514 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:23.896383    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1312ccd3-4c59-4cd1-b087-93c9aabcd152-kube-api-access-ggzsc\" (UniqueName: \"kubernetes.io/projected/1312ccd3-4c59-4cd1-b087-93c9aabcd152-kube-api-access-ggzsc\") pod \"1312ccd3-4c59-4cd1-b087-93c9aabcd152\" (UID: \"1312ccd3-4c59-4cd1-b087-93c9aabcd152\") "
Nov 01 06:31:23.900523 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:23.899719    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1312ccd3-4c59-4cd1-b087-93c9aabcd152-openshift-service-ca" pod "1312ccd3-4c59-4cd1-b087-93c9aabcd152" (UID: "1312ccd3-4c59-4cd1-b087-93c9aabcd152"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:31:23.908792 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:23.908730    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1312ccd3-4c59-4cd1-b087-93c9aabcd152-kube-api-access-ggzsc" pod "1312ccd3-4c59-4cd1-b087-93c9aabcd152" (UID: "1312ccd3-4c59-4cd1-b087-93c9aabcd152"). InnerVolumeSpecName "kube-api-access-ggzsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:31:23.908792 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:23.908741    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1312ccd3-4c59-4cd1-b087-93c9aabcd152-maas-api-tls" pod "1312ccd3-4c59-4cd1-b087-93c9aabcd152" (UID: "1312ccd3-4c59-4cd1-b087-93c9aabcd152"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:31:23.910679 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:23.910620    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1312ccd3-4c59-4cd1-b087-93c9aabcd152-metrics-certs" pod "1312ccd3-4c59-4cd1-b087-93c9aabcd152" (UID: "1312ccd3-4c59-4cd1-b087-93c9aabcd152"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:31:23.998642 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:23.998612    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1312ccd3-4c59-4cd1-b087-93c9aabcd152-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:23.998642 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:23.998640    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1312ccd3-4c59-4cd1-b087-93c9aabcd152-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:23.998642 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:23.998649    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1312ccd3-4c59-4cd1-b087-93c9aabcd152-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:23.998875 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:23.998657    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggzsc\" (UniqueName: \"kubernetes.io/projected/1312ccd3-4c59-4cd1-b087-93c9aabcd152-kube-api-access-ggzsc\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:31:24.510548 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:24.510513    5619 generic.go:356] "Generic (PLEG): container finished" podID="1312ccd3-4c59-4cd1-b087-93c9aabcd152" containerID="3fd48e9d111882b79078c9558af5a0a2b54fa5423e73f8561d01cce02754e8c3" exitCode=0
Nov 01 06:31:24.510953 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:24.510564    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"
Nov 01 06:31:24.510953 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:24.510604    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt" event={"ID":"1312ccd3-4c59-4cd1-b087-93c9aabcd152","Type":"ContainerDied","Data":"3fd48e9d111882b79078c9558af5a0a2b54fa5423e73f8561d01cce02754e8c3"}
Nov 01 06:31:24.510953 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:24.510660    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt" event={"ID":"1312ccd3-4c59-4cd1-b087-93c9aabcd152","Type":"ContainerDied","Data":"7f5f94139f281e6ebb010a87a76a60a32d992e4edfa83a6a10a23eec895f325f"}
Nov 01 06:31:24.510953 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:24.510691    5619 scope.go:122] "RemoveContainer" containerID="3fd48e9d111882b79078c9558af5a0a2b54fa5423e73f8561d01cce02754e8c3"
Nov 01 06:31:24.520739 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:24.520719    5619 scope.go:122] "RemoveContainer" containerID="3fd48e9d111882b79078c9558af5a0a2b54fa5423e73f8561d01cce02754e8c3"
Nov 01 06:31:24.521001 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:31:24.520983    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fd48e9d111882b79078c9558af5a0a2b54fa5423e73f8561d01cce02754e8c3\": container with ID starting with 3fd48e9d111882b79078c9558af5a0a2b54fa5423e73f8561d01cce02754e8c3 not found: ID does not exist" containerID="3fd48e9d111882b79078c9558af5a0a2b54fa5423e73f8561d01cce02754e8c3"
Nov 01 06:31:24.521056 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:24.521010    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fd48e9d111882b79078c9558af5a0a2b54fa5423e73f8561d01cce02754e8c3"} err="failed to get container status \"3fd48e9d111882b79078c9558af5a0a2b54fa5423e73f8561d01cce02754e8c3\": rpc error: code = NotFound desc = could not find container \"3fd48e9d111882b79078c9558af5a0a2b54fa5423e73f8561d01cce02754e8c3\": container with ID starting with 3fd48e9d111882b79078c9558af5a0a2b54fa5423e73f8561d01cce02754e8c3 not found: ID does not exist"
Nov 01 06:31:24.680746 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:24.680709    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"]
Nov 01 06:31:24.729891 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:24.729859    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f13905-859955f999-824pt"]
Nov 01 06:31:24.963419 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:24.963388    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1312ccd3-4c59-4cd1-b087-93c9aabcd152" path="/var/lib/kubelet/pods/1312ccd3-4c59-4cd1-b087-93c9aabcd152/volumes"
Nov 01 06:31:25.363068 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:25.363025    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:25.880658 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:31:25.880625    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:31:26.131366 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:26.131257    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:27.102557 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:27.102523    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:30.064004 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:30.063966    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:30.533004 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:30.532971    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:30.682493 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:30.682446    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:31.826337 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:31.826286    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:34.757993 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:34.757955    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:39.508063 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:39.508026    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:42.090001 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:42.089965    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:42.608375 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:42.608312    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:43.371306 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:43.371261    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:44.442420 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:44.442378    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:49.411586 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:49.411547    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:49.605588 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:49.605552    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:50.187824 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:50.187788    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:50.326750 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:50.326715    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:51.914975 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:51.914937    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:52.109896 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:52.109848    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:54.650078 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:54.650037    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:55.797738 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:55.797633    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:55.882582 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:31:55.882537    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:31:56.010823 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:56.009338    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv"]
Nov 01 06:31:56.266085 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:56.266040    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:56.407227 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:56.407187    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:31:58.723377 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:31:58.723318    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:00.432409 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:00.432371    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:01.010770 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:01.010729    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv" podUID="a43f8f69-2a4e-40bc-8f81-154aef4e9312" containerName="payload-pre-processing" containerID="cri-o://e9ec7f424bb7cbf1f26df7e012c9f8f4d3944d31834b75b2116d68b5ca8f201a" gracePeriod=25
Nov 01 06:32:01.275782 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:01.275718    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv"
Nov 01 06:32:01.374294 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:01.374257    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a43f8f69-2a4e-40bc-8f81-154aef4e9312-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43f8f69-2a4e-40bc-8f81-154aef4e9312-plugins-config-volume\") pod \"a43f8f69-2a4e-40bc-8f81-154aef4e9312\" (UID: \"a43f8f69-2a4e-40bc-8f81-154aef4e9312\") "
Nov 01 06:32:01.374515 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:01.374494    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a43f8f69-2a4e-40bc-8f81-154aef4e9312-kube-api-access-bbtmg\" (UniqueName: \"kubernetes.io/projected/a43f8f69-2a4e-40bc-8f81-154aef4e9312-kube-api-access-bbtmg\") pod \"a43f8f69-2a4e-40bc-8f81-154aef4e9312\" (UID: \"a43f8f69-2a4e-40bc-8f81-154aef4e9312\") "
Nov 01 06:32:01.374745 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:01.374723    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a43f8f69-2a4e-40bc-8f81-154aef4e9312-plugins-config-volume" pod "a43f8f69-2a4e-40bc-8f81-154aef4e9312" (UID: "a43f8f69-2a4e-40bc-8f81-154aef4e9312"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:32:01.374963 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:01.374948    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43f8f69-2a4e-40bc-8f81-154aef4e9312-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:32:01.376608 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:01.376584    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43f8f69-2a4e-40bc-8f81-154aef4e9312-kube-api-access-bbtmg" pod "a43f8f69-2a4e-40bc-8f81-154aef4e9312" (UID: "a43f8f69-2a4e-40bc-8f81-154aef4e9312"). InnerVolumeSpecName "kube-api-access-bbtmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:32:01.476418 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:01.476378    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbtmg\" (UniqueName: \"kubernetes.io/projected/a43f8f69-2a4e-40bc-8f81-154aef4e9312-kube-api-access-bbtmg\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:32:01.764442 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:01.764406    5619 generic.go:356] "Generic (PLEG): container finished" podID="a43f8f69-2a4e-40bc-8f81-154aef4e9312" containerID="e9ec7f424bb7cbf1f26df7e012c9f8f4d3944d31834b75b2116d68b5ca8f201a" exitCode=0
Nov 01 06:32:01.764633 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:01.764459    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv"
Nov 01 06:32:01.764633 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:01.764498    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv" event={"ID":"a43f8f69-2a4e-40bc-8f81-154aef4e9312","Type":"ContainerDied","Data":"e9ec7f424bb7cbf1f26df7e012c9f8f4d3944d31834b75b2116d68b5ca8f201a"}
Nov 01 06:32:01.764633 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:01.764552    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv" event={"ID":"a43f8f69-2a4e-40bc-8f81-154aef4e9312","Type":"ContainerDied","Data":"7b65d022589acb3e73ee7fdbb73706d626a9d71ca303dffcfd0cbcda864c40ea"}
Nov 01 06:32:01.764633 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:01.764576    5619 scope.go:122] "RemoveContainer" containerID="e9ec7f424bb7cbf1f26df7e012c9f8f4d3944d31834b75b2116d68b5ca8f201a"
Nov 01 06:32:01.774864 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:01.774847    5619 scope.go:122] "RemoveContainer" containerID="e9ec7f424bb7cbf1f26df7e012c9f8f4d3944d31834b75b2116d68b5ca8f201a"
Nov 01 06:32:01.775132 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:32:01.775113    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ec7f424bb7cbf1f26df7e012c9f8f4d3944d31834b75b2116d68b5ca8f201a\": container with ID starting with e9ec7f424bb7cbf1f26df7e012c9f8f4d3944d31834b75b2116d68b5ca8f201a not found: ID does not exist" containerID="e9ec7f424bb7cbf1f26df7e012c9f8f4d3944d31834b75b2116d68b5ca8f201a"
Nov 01 06:32:01.775179 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:01.775139    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ec7f424bb7cbf1f26df7e012c9f8f4d3944d31834b75b2116d68b5ca8f201a"} err="failed to get container status \"e9ec7f424bb7cbf1f26df7e012c9f8f4d3944d31834b75b2116d68b5ca8f201a\": rpc error: code = NotFound desc = could not find container \"e9ec7f424bb7cbf1f26df7e012c9f8f4d3944d31834b75b2116d68b5ca8f201a\": container with ID starting with e9ec7f424bb7cbf1f26df7e012c9f8f4d3944d31834b75b2116d68b5ca8f201a not found: ID does not exist"
Nov 01 06:32:01.838089 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:01.838056    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:01.931840 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:01.931807    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv"]
Nov 01 06:32:01.982444 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:01.982404    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0aed62e2-7597445fbc-22mgv"]
Nov 01 06:32:02.027204 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:02.027117    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:02.962459 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:02.962429    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a43f8f69-2a4e-40bc-8f81-154aef4e9312" path="/var/lib/kubelet/pods/a43f8f69-2a4e-40bc-8f81-154aef4e9312/volumes"
Nov 01 06:32:03.358843 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:03.358807    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:03.841292 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:03.841257    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:05.310423 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:05.310372    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:10.306123 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:10.306079    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:10.463578 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:10.463536    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:10.603085 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:10.603049    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:14.736662 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:14.736622    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:15.243948 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:15.243894    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:15.386221 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:15.386180    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:15.874156 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:15.874119    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:19.615232 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:19.615174    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:19.809434 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:19.809394    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:20.232804 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:20.232770    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:22.809380 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:22.809311    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:22.983862 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:22.983821    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:23.214514 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.214472    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw"]
Nov 01 06:32:23.215180 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.215157    5619 cpu_manager.go:414] "containerMap: removing container" podUID="a43f8f69-2a4e-40bc-8f81-154aef4e9312" containerName="payload-pre-processing"
Nov 01 06:32:23.215287 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.215182    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a43f8f69-2a4e-40bc-8f81-154aef4e9312" containerName="payload-pre-processing"
Nov 01 06:32:23.215287 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.215229    5619 cpu_manager.go:414] "containerMap: removing container" podUID="1312ccd3-4c59-4cd1-b087-93c9aabcd152" containerName="maas-api"
Nov 01 06:32:23.215287 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.215237    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1312ccd3-4c59-4cd1-b087-93c9aabcd152" containerName="maas-api"
Nov 01 06:32:23.215492 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.215409    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="a43f8f69-2a4e-40bc-8f81-154aef4e9312" containerName="payload-pre-processing"
Nov 01 06:32:23.215492 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.215430    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="1312ccd3-4c59-4cd1-b087-93c9aabcd152" containerName="maas-api"
Nov 01 06:32:23.220071 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.220049    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw"
Nov 01 06:32:23.244532 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.244498    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4nrv\" (UniqueName: \"kubernetes.io/projected/c86eab09-9ed0-4dc3-9b5e-65b59abef247-kube-api-access-q4nrv\") pod \"payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw\" (UID: \"c86eab09-9ed0-4dc3-9b5e-65b59abef247\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw"
Nov 01 06:32:23.244694 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.244609    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86eab09-9ed0-4dc3-9b5e-65b59abef247-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw\" (UID: \"c86eab09-9ed0-4dc3-9b5e-65b59abef247\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw"
Nov 01 06:32:23.277167 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.277129    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw"]
Nov 01 06:32:23.342780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.342731    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-3680c5\""
Nov 01 06:32:23.343206 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.343176    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-3680c5-dockercfg-f95xx\""
Nov 01 06:32:23.365364 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.358015    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86eab09-9ed0-4dc3-9b5e-65b59abef247-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw\" (UID: \"c86eab09-9ed0-4dc3-9b5e-65b59abef247\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw"
Nov 01 06:32:23.365364 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.358253    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4nrv\" (UniqueName: \"kubernetes.io/projected/c86eab09-9ed0-4dc3-9b5e-65b59abef247-kube-api-access-q4nrv\") pod \"payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw\" (UID: \"c86eab09-9ed0-4dc3-9b5e-65b59abef247\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw"
Nov 01 06:32:23.365364 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.363894    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86eab09-9ed0-4dc3-9b5e-65b59abef247-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw\" (UID: \"c86eab09-9ed0-4dc3-9b5e-65b59abef247\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw"
Nov 01 06:32:23.386999 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.386917    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67"]
Nov 01 06:32:23.392487 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.392455    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67"
Nov 01 06:32:23.434067 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.434037    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4nrv\" (UniqueName: \"kubernetes.io/projected/c86eab09-9ed0-4dc3-9b5e-65b59abef247-kube-api-access-q4nrv\") pod \"payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw\" (UID: \"c86eab09-9ed0-4dc3-9b5e-65b59abef247\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw"
Nov 01 06:32:23.446364 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.446317    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67"]
Nov 01 06:32:23.470406 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.470287    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c20bff8c-c80c-4776-87d1-7dc271ae1dc7-plugins-config-volume\") pod \"payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67\" (UID: \"c20bff8c-c80c-4776-87d1-7dc271ae1dc7\") " pod="openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67"
Nov 01 06:32:23.470629 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.470484    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74flp\" (UniqueName: \"kubernetes.io/projected/c20bff8c-c80c-4776-87d1-7dc271ae1dc7-kube-api-access-74flp\") pod \"payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67\" (UID: \"c20bff8c-c80c-4776-87d1-7dc271ae1dc7\") " pod="openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67"
Nov 01 06:32:23.533249 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.533213    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw"
Nov 01 06:32:23.574369 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.571473    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c20bff8c-c80c-4776-87d1-7dc271ae1dc7-plugins-config-volume\") pod \"payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67\" (UID: \"c20bff8c-c80c-4776-87d1-7dc271ae1dc7\") " pod="openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67"
Nov 01 06:32:23.574369 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.571804    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74flp\" (UniqueName: \"kubernetes.io/projected/c20bff8c-c80c-4776-87d1-7dc271ae1dc7-kube-api-access-74flp\") pod \"payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67\" (UID: \"c20bff8c-c80c-4776-87d1-7dc271ae1dc7\") " pod="openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67"
Nov 01 06:32:23.574369 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.572465    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c20bff8c-c80c-4776-87d1-7dc271ae1dc7-plugins-config-volume\") pod \"payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67\" (UID: \"c20bff8c-c80c-4776-87d1-7dc271ae1dc7\") " pod="openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67"
Nov 01 06:32:23.635422 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.635388    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74flp\" (UniqueName: \"kubernetes.io/projected/c20bff8c-c80c-4776-87d1-7dc271ae1dc7-kube-api-access-74flp\") pod \"payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67\" (UID: \"c20bff8c-c80c-4776-87d1-7dc271ae1dc7\") " pod="openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67"
Nov 01 06:32:23.711290 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.711259    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67"
Nov 01 06:32:23.926468 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:23.926433    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw"]
Nov 01 06:32:23.927619 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:32:23.927590    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc86eab09_9ed0_4dc3_9b5e_65b59abef247.slice/crio-e2f49b54ca92faa0b51274f0cc7476c08030baf4150f5bceb86557347c0437c8 WatchSource:0}: Error finding container e2f49b54ca92faa0b51274f0cc7476c08030baf4150f5bceb86557347c0437c8: Status 404 returned error can't find the container with id e2f49b54ca92faa0b51274f0cc7476c08030baf4150f5bceb86557347c0437c8
Nov 01 06:32:24.103317 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:24.103281    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67"]
Nov 01 06:32:24.917216 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:24.917173    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67" event={"ID":"c20bff8c-c80c-4776-87d1-7dc271ae1dc7","Type":"ContainerStarted","Data":"2e2b7b5926ee71cb7ea04333d0a192ff6609d502ab5534d3cc78780ca536c72d"}
Nov 01 06:32:24.917471 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:24.917243    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67" event={"ID":"c20bff8c-c80c-4776-87d1-7dc271ae1dc7","Type":"ContainerStarted","Data":"80aa4ea35b7e73648c3fc40d4bb18948ebba766e4414221f08a51195e8346c4a"}
Nov 01 06:32:24.917471 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:24.917280    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67"
Nov 01 06:32:24.919081 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:24.919049    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw" event={"ID":"c86eab09-9ed0-4dc3-9b5e-65b59abef247","Type":"ContainerStarted","Data":"f03b8ba0561ce12d9c2cd821f7a40dded972a9df9d225847522bdf34bef34d71"}
Nov 01 06:32:24.919209 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:24.919102    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw" event={"ID":"c86eab09-9ed0-4dc3-9b5e-65b59abef247","Type":"ContainerStarted","Data":"e2f49b54ca92faa0b51274f0cc7476c08030baf4150f5bceb86557347c0437c8"}
Nov 01 06:32:24.919209 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:24.919131    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw"
Nov 01 06:32:25.039234 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:25.039160    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67" podStartSLOduration=2.039140683 podStartE2EDuration="2.039140683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:32:25.037264415 +0000 UTC m=+2782.248087938" watchObservedRunningTime="2026-11-01 06:32:25.039140683 +0000 UTC m=+2782.249964199"
Nov 01 06:32:25.158359 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:25.158287    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw" podStartSLOduration=2.158270583 podStartE2EDuration="2.158270583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:32:25.154724988 +0000 UTC m=+2782.365548540" watchObservedRunningTime="2026-11-01 06:32:25.158270583 +0000 UTC m=+2782.369094091"
Nov 01 06:32:25.684198 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:25.684160    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:25.883207 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:32:25.883172    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:32:27.505550 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:27.505519    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:30.479713 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:30.479672    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:32.167855 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:32.167815    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:35.251708 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:35.251671    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:35.928474 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:35.928442    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67"
Nov 01 06:32:35.928721 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:35.928533    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw"
Nov 01 06:32:39.973214 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:39.973169    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:44.756183 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:44.756075    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:49.564297 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:49.564261    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:51.234862 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:51.234824    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:54.360110 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:54.360077    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:32:55.880829 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:32:55.880789    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:32:59.059177 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:32:59.059136    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:00.534685 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:00.534637    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:00.726965 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:00.726929    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:01.888487 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:01.888447    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"]
Nov 01 06:33:01.889024 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:01.888709    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm" podUID="b44c28d4-b7ca-4350-b78a-53e4c9036124" containerName="istio-proxy" containerID="cri-o://922fea25a0e6087c22d7893da1c361252c89c5db7f3fa0d25b33306e81f25b25" gracePeriod=30
Nov 01 06:33:02.399891 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:02.399857    5619 patch_prober.go:29] interesting pod/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:33:02.400065 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:02.399917    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm" podUID="b44c28d4-b7ca-4350-b78a-53e4c9036124" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:33:03.109598 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:03.109563    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"]
Nov 01 06:33:03.110028 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:03.109788    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7" podUID="5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8" containerName="maas-api" containerID="cri-o://07faa4161b816bd276187d15f5bc62caf24c5f11d835b1b5f7db4967fd1e11fb" gracePeriod=30
Nov 01 06:33:03.204014 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:03.203981    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:03.239157 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:03.239115    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7" podUID="5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.126:8443/health\": dial tcp 10.133.0.126:8443: connect: connection refused"
Nov 01 06:33:03.400798 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:03.400768    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"
Nov 01 06:33:03.572141 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:03.572108    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-maas-api-tls\") pod \"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8\" (UID: \"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8\") "
Nov 01 06:33:03.572408 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:03.572176    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-kube-api-access-pcvms\" (UniqueName: \"kubernetes.io/projected/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-kube-api-access-pcvms\") pod \"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8\" (UID: \"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8\") "
Nov 01 06:33:03.572408 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:03.572217    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-metrics-certs\") pod \"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8\" (UID: \"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8\") "
Nov 01 06:33:03.572408 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:03.572350    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-openshift-service-ca\") pod \"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8\" (UID: \"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8\") "
Nov 01 06:33:03.572744 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:03.572714    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-openshift-service-ca" pod "5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8" (UID: "5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:33:03.573017 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:03.572996    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:03.574736 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:03.574695    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-metrics-certs" pod "5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8" (UID: "5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:33:03.574960 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:03.574939    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-maas-api-tls" pod "5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8" (UID: "5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:33:03.575075 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:03.575056    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-kube-api-access-pcvms" pod "5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8" (UID: "5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8"). InnerVolumeSpecName "kube-api-access-pcvms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:33:03.674622 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:03.674537    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:03.674622 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:03.674571    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcvms\" (UniqueName: \"kubernetes.io/projected/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-kube-api-access-pcvms\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:03.674622 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:03.674583    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:04.169676 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.169645    5619 generic.go:356] "Generic (PLEG): container finished" podID="5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8" containerID="07faa4161b816bd276187d15f5bc62caf24c5f11d835b1b5f7db4967fd1e11fb" exitCode=0
Nov 01 06:33:04.170171 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.169691    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"
Nov 01 06:33:04.170171 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.169725    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7" event={"ID":"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8","Type":"ContainerDied","Data":"07faa4161b816bd276187d15f5bc62caf24c5f11d835b1b5f7db4967fd1e11fb"}
Nov 01 06:33:04.170171 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.169776    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7" event={"ID":"5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8","Type":"ContainerDied","Data":"20a5199ae4cdb859d392e67239b2bcc51aa222f1100a96ee7d10d6f5fffff011"}
Nov 01 06:33:04.170171 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.169804    5619 scope.go:122] "RemoveContainer" containerID="07faa4161b816bd276187d15f5bc62caf24c5f11d835b1b5f7db4967fd1e11fb"
Nov 01 06:33:04.184317 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.183985    5619 scope.go:122] "RemoveContainer" containerID="07faa4161b816bd276187d15f5bc62caf24c5f11d835b1b5f7db4967fd1e11fb"
Nov 01 06:33:04.184630 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:33:04.184604    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07faa4161b816bd276187d15f5bc62caf24c5f11d835b1b5f7db4967fd1e11fb\": container with ID starting with 07faa4161b816bd276187d15f5bc62caf24c5f11d835b1b5f7db4967fd1e11fb not found: ID does not exist" containerID="07faa4161b816bd276187d15f5bc62caf24c5f11d835b1b5f7db4967fd1e11fb"
Nov 01 06:33:04.184706 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.184638    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07faa4161b816bd276187d15f5bc62caf24c5f11d835b1b5f7db4967fd1e11fb"} err="failed to get container status \"07faa4161b816bd276187d15f5bc62caf24c5f11d835b1b5f7db4967fd1e11fb\": rpc error: code = NotFound desc = could not find container \"07faa4161b816bd276187d15f5bc62caf24c5f11d835b1b5f7db4967fd1e11fb\": container with ID starting with 07faa4161b816bd276187d15f5bc62caf24c5f11d835b1b5f7db4967fd1e11fb not found: ID does not exist"
Nov 01 06:33:04.342019 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.341983    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"]
Nov 01 06:33:04.392822 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.392791    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0aed62e2-7679c95c77-tjjj7"]
Nov 01 06:33:04.684446 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.684409    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:04.766180 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.766140    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"]
Nov 01 06:33:04.767040 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.767015    5619 cpu_manager.go:414] "containerMap: removing container" podUID="5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8" containerName="maas-api"
Nov 01 06:33:04.767166 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.767042    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8" containerName="maas-api"
Nov 01 06:33:04.767265 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.767231    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8" containerName="maas-api"
Nov 01 06:33:04.772678 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.772654    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"
Nov 01 06:33:04.826089 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.826060    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"agc-cred-31d2d9550e36\""
Nov 01 06:33:04.829069 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.829043    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"]
Nov 01 06:33:04.887934 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.887896    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l79n\" (UniqueName: \"kubernetes.io/projected/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-kube-api-access-5l79n\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj\" (UID: \"e98c4ffb-6a21-459d-be1b-0bde6489e0c1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"
Nov 01 06:33:04.888097 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.887963    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj\" (UID: \"e98c4ffb-6a21-459d-be1b-0bde6489e0c1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"
Nov 01 06:33:04.888145 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.888120    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj\" (UID: \"e98c4ffb-6a21-459d-be1b-0bde6489e0c1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"
Nov 01 06:33:04.888264 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.888243    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj\" (UID: \"e98c4ffb-6a21-459d-be1b-0bde6489e0c1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"
Nov 01 06:33:04.911979 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.911947    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:04.965282 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.965193    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8" path="/var/lib/kubelet/pods/5b8834a9-3d3d-41da-bd8e-2b8cb85c74d8/volumes"
Nov 01 06:33:04.989524 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.989490    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj\" (UID: \"e98c4ffb-6a21-459d-be1b-0bde6489e0c1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"
Nov 01 06:33:04.989698 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.989566    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj\" (UID: \"e98c4ffb-6a21-459d-be1b-0bde6489e0c1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"
Nov 01 06:33:04.989698 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.989621    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l79n\" (UniqueName: \"kubernetes.io/projected/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-kube-api-access-5l79n\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj\" (UID: \"e98c4ffb-6a21-459d-be1b-0bde6489e0c1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"
Nov 01 06:33:04.989698 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.989666    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj\" (UID: \"e98c4ffb-6a21-459d-be1b-0bde6489e0c1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"
Nov 01 06:33:04.990385 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.990356    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj\" (UID: \"e98c4ffb-6a21-459d-be1b-0bde6489e0c1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"
Nov 01 06:33:04.990493 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.990400    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj\" (UID: \"e98c4ffb-6a21-459d-be1b-0bde6489e0c1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"
Nov 01 06:33:04.992347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:04.992309    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj\" (UID: \"e98c4ffb-6a21-459d-be1b-0bde6489e0c1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"
Nov 01 06:33:05.052407 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:05.052372    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l79n\" (UniqueName: \"kubernetes.io/projected/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-kube-api-access-5l79n\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj\" (UID: \"e98c4ffb-6a21-459d-be1b-0bde6489e0c1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"
Nov 01 06:33:05.053789 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:05.053760    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:05.087193 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:05.087163    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"
Nov 01 06:33:05.281696 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:05.281611    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:05.469159 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:33:05.469125    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode98c4ffb_6a21_459d_be1b_0bde6489e0c1.slice/crio-9e18d0a54b631fde67252c7fd2f467d3957d09fd9a61a46a13de9d70e272bc30 WatchSource:0}: Error finding container 9e18d0a54b631fde67252c7fd2f467d3957d09fd9a61a46a13de9d70e272bc30: Status 404 returned error can't find the container with id 9e18d0a54b631fde67252c7fd2f467d3957d09fd9a61a46a13de9d70e272bc30
Nov 01 06:33:05.470713 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:05.470685    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"]
Nov 01 06:33:06.189031 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:06.188990    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj" event={"ID":"e98c4ffb-6a21-459d-be1b-0bde6489e0c1","Type":"ContainerStarted","Data":"009aa7ec8853dd5bdac1828bc491ab34771daa018926c4b165e17b7aea07803a"}
Nov 01 06:33:06.189238 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:06.189056    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj" event={"ID":"e98c4ffb-6a21-459d-be1b-0bde6489e0c1","Type":"ContainerStarted","Data":"9e18d0a54b631fde67252c7fd2f467d3957d09fd9a61a46a13de9d70e272bc30"}
Nov 01 06:33:06.189238 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:06.189214    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"
Nov 01 06:33:06.307513 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:06.307462    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj" podStartSLOduration=2.307447439 podStartE2EDuration="2.307447439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:33:06.306645429 +0000 UTC m=+2823.517468929" watchObservedRunningTime="2026-11-01 06:33:06.307447439 +0000 UTC m=+2823.518270948"
Nov 01 06:33:06.565350 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:06.565251    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:06.753757 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:06.753715    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:07.164652 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.164590    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:33:07.196741 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.196708    5619 generic.go:356] "Generic (PLEG): container finished" podID="b44c28d4-b7ca-4350-b78a-53e4c9036124" containerID="922fea25a0e6087c22d7893da1c361252c89c5db7f3fa0d25b33306e81f25b25" exitCode=0
Nov 01 06:33:07.196906 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.196761    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"
Nov 01 06:33:07.196906 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.196800    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm" event={"ID":"b44c28d4-b7ca-4350-b78a-53e4c9036124","Type":"ContainerDied","Data":"922fea25a0e6087c22d7893da1c361252c89c5db7f3fa0d25b33306e81f25b25"}
Nov 01 06:33:07.196906 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.196869    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm" event={"ID":"b44c28d4-b7ca-4350-b78a-53e4c9036124","Type":"ContainerDied","Data":"8d35e306228c017abb5f458818542431d0bb3938434d040e156792aef5b089c7"}
Nov 01 06:33:07.196906 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.196889    5619 scope.go:122] "RemoveContainer" containerID="922fea25a0e6087c22d7893da1c361252c89c5db7f3fa0d25b33306e81f25b25"
Nov 01 06:33:07.206818 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.206799    5619 scope.go:122] "RemoveContainer" containerID="922fea25a0e6087c22d7893da1c361252c89c5db7f3fa0d25b33306e81f25b25"
Nov 01 06:33:07.207103 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:33:07.207083    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"922fea25a0e6087c22d7893da1c361252c89c5db7f3fa0d25b33306e81f25b25\": container with ID starting with 922fea25a0e6087c22d7893da1c361252c89c5db7f3fa0d25b33306e81f25b25 not found: ID does not exist" containerID="922fea25a0e6087c22d7893da1c361252c89c5db7f3fa0d25b33306e81f25b25"
Nov 01 06:33:07.207175 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.207109    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"922fea25a0e6087c22d7893da1c361252c89c5db7f3fa0d25b33306e81f25b25"} err="failed to get container status \"922fea25a0e6087c22d7893da1c361252c89c5db7f3fa0d25b33306e81f25b25\": rpc error: code = NotFound desc = could not find container \"922fea25a0e6087c22d7893da1c361252c89c5db7f3fa0d25b33306e81f25b25\": container with ID starting with 922fea25a0e6087c22d7893da1c361252c89c5db7f3fa0d25b33306e81f25b25 not found: ID does not exist"
Nov 01 06:33:07.213612 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.213593    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b44c28d4-b7ca-4350-b78a-53e4c9036124-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44c28d4-b7ca-4350-b78a-53e4c9036124-istiod-ca-cert\") pod \"b44c28d4-b7ca-4350-b78a-53e4c9036124\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") "
Nov 01 06:33:07.213704 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.213627    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-credential-socket\") pod \"b44c28d4-b7ca-4350-b78a-53e4c9036124\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") "
Nov 01 06:33:07.213704 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.213655    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-envoy\") pod \"b44c28d4-b7ca-4350-b78a-53e4c9036124\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") "
Nov 01 06:33:07.213704 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.213683    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b44c28d4-b7ca-4350-b78a-53e4c9036124-kube-api-access-mxk9v\" (UniqueName: \"kubernetes.io/projected/b44c28d4-b7ca-4350-b78a-53e4c9036124-kube-api-access-mxk9v\") pod \"b44c28d4-b7ca-4350-b78a-53e4c9036124\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") "
Nov 01 06:33:07.213862 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.213720    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-podinfo\") pod \"b44c28d4-b7ca-4350-b78a-53e4c9036124\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") "
Nov 01 06:33:07.213862 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.213767    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-data\") pod \"b44c28d4-b7ca-4350-b78a-53e4c9036124\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") "
Nov 01 06:33:07.213862 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.213832    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-workload-socket\") pod \"b44c28d4-b7ca-4350-b78a-53e4c9036124\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") "
Nov 01 06:33:07.214024 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.213895    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-workload-certs\") pod \"b44c28d4-b7ca-4350-b78a-53e4c9036124\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") "
Nov 01 06:33:07.214024 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.213926    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-credential-socket" pod "b44c28d4-b7ca-4350-b78a-53e4c9036124" (UID: "b44c28d4-b7ca-4350-b78a-53e4c9036124"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:33:07.214024 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.213952    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-token\" (UniqueName: \"kubernetes.io/projected/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-token\") pod \"b44c28d4-b7ca-4350-b78a-53e4c9036124\" (UID: \"b44c28d4-b7ca-4350-b78a-53e4c9036124\") "
Nov 01 06:33:07.214219 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.213950    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44c28d4-b7ca-4350-b78a-53e4c9036124-istiod-ca-cert" pod "b44c28d4-b7ca-4350-b78a-53e4c9036124" (UID: "b44c28d4-b7ca-4350-b78a-53e4c9036124"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:33:07.214219 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.214179    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-data" pod "b44c28d4-b7ca-4350-b78a-53e4c9036124" (UID: "b44c28d4-b7ca-4350-b78a-53e4c9036124"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:33:07.214342 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.214195    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-workload-certs" pod "b44c28d4-b7ca-4350-b78a-53e4c9036124" (UID: "b44c28d4-b7ca-4350-b78a-53e4c9036124"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:33:07.214426 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.214399    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-workload-socket" pod "b44c28d4-b7ca-4350-b78a-53e4c9036124" (UID: "b44c28d4-b7ca-4350-b78a-53e4c9036124"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:33:07.214761 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.214715    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44c28d4-b7ca-4350-b78a-53e4c9036124-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:07.214761 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.214742    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:07.214761 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.214758    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:07.214975 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.214773    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:07.214975 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.214789    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:07.216168 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.216139    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44c28d4-b7ca-4350-b78a-53e4c9036124-kube-api-access-mxk9v" pod "b44c28d4-b7ca-4350-b78a-53e4c9036124" (UID: "b44c28d4-b7ca-4350-b78a-53e4c9036124"). InnerVolumeSpecName "kube-api-access-mxk9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:33:07.216486 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.216465    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-envoy" pod "b44c28d4-b7ca-4350-b78a-53e4c9036124" (UID: "b44c28d4-b7ca-4350-b78a-53e4c9036124"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:33:07.216555 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.216527    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-token" pod "b44c28d4-b7ca-4350-b78a-53e4c9036124" (UID: "b44c28d4-b7ca-4350-b78a-53e4c9036124"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:33:07.216677 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.216656    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-podinfo" pod "b44c28d4-b7ca-4350-b78a-53e4c9036124" (UID: "b44c28d4-b7ca-4350-b78a-53e4c9036124"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:33:07.316302 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.316257    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxk9v\" (UniqueName: \"kubernetes.io/projected/b44c28d4-b7ca-4350-b78a-53e4c9036124-kube-api-access-mxk9v\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:07.316302 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.316293    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:07.316870 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.316310    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:07.316870 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.316338    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44c28d4-b7ca-4350-b78a-53e4c9036124-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:07.629020 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.628981    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"]
Nov 01 06:33:07.660583 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:07.660545    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-7qlwm"]
Nov 01 06:33:08.963159 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:08.963125    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b44c28d4-b7ca-4350-b78a-53e4c9036124" path="/var/lib/kubelet/pods/b44c28d4-b7ca-4350-b78a-53e4c9036124/volumes"
Nov 01 06:33:09.860933 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:09.860891    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:09.872598 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:09.872573    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"]
Nov 01 06:33:09.923552 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:09.923519    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"]
Nov 01 06:33:09.924313 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:09.924292    5619 cpu_manager.go:414] "containerMap: removing container" podUID="b44c28d4-b7ca-4350-b78a-53e4c9036124" containerName="istio-proxy"
Nov 01 06:33:09.924382 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:09.924316    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44c28d4-b7ca-4350-b78a-53e4c9036124" containerName="istio-proxy"
Nov 01 06:33:09.924478 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:09.924467    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44c28d4-b7ca-4350-b78a-53e4c9036124" containerName="istio-proxy"
Nov 01 06:33:09.929722 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:09.929702    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"
Nov 01 06:33:09.983051 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:09.983026    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"agc-cred-e14e279743a3\""
Nov 01 06:33:09.995844 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:09.995808    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"]
Nov 01 06:33:10.068138 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:10.068106    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csxjx\" (UniqueName: \"kubernetes.io/projected/75319d6b-af9b-4c2f-a53d-71b2a64f3325-kube-api-access-csxjx\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb\" (UID: \"75319d6b-af9b-4c2f-a53d-71b2a64f3325\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"
Nov 01 06:33:10.068307 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:10.068190    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75319d6b-af9b-4c2f-a53d-71b2a64f3325-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb\" (UID: \"75319d6b-af9b-4c2f-a53d-71b2a64f3325\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"
Nov 01 06:33:10.068451 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:10.068427    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75319d6b-af9b-4c2f-a53d-71b2a64f3325-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb\" (UID: \"75319d6b-af9b-4c2f-a53d-71b2a64f3325\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"
Nov 01 06:33:10.068578 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:10.068555    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75319d6b-af9b-4c2f-a53d-71b2a64f3325-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb\" (UID: \"75319d6b-af9b-4c2f-a53d-71b2a64f3325\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"
Nov 01 06:33:10.170417 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:10.170318    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75319d6b-af9b-4c2f-a53d-71b2a64f3325-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb\" (UID: \"75319d6b-af9b-4c2f-a53d-71b2a64f3325\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"
Nov 01 06:33:10.170545 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:10.170455    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csxjx\" (UniqueName: \"kubernetes.io/projected/75319d6b-af9b-4c2f-a53d-71b2a64f3325-kube-api-access-csxjx\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb\" (UID: \"75319d6b-af9b-4c2f-a53d-71b2a64f3325\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"
Nov 01 06:33:10.170545 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:10.170512    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75319d6b-af9b-4c2f-a53d-71b2a64f3325-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb\" (UID: \"75319d6b-af9b-4c2f-a53d-71b2a64f3325\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"
Nov 01 06:33:10.170612 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:10.170575    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75319d6b-af9b-4c2f-a53d-71b2a64f3325-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb\" (UID: \"75319d6b-af9b-4c2f-a53d-71b2a64f3325\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"
Nov 01 06:33:10.170992 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:10.170961    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75319d6b-af9b-4c2f-a53d-71b2a64f3325-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb\" (UID: \"75319d6b-af9b-4c2f-a53d-71b2a64f3325\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"
Nov 01 06:33:10.171314 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:10.171294    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75319d6b-af9b-4c2f-a53d-71b2a64f3325-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb\" (UID: \"75319d6b-af9b-4c2f-a53d-71b2a64f3325\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"
Nov 01 06:33:10.173316 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:10.173293    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75319d6b-af9b-4c2f-a53d-71b2a64f3325-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb\" (UID: \"75319d6b-af9b-4c2f-a53d-71b2a64f3325\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"
Nov 01 06:33:10.232315 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:10.232283    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csxjx\" (UniqueName: \"kubernetes.io/projected/75319d6b-af9b-4c2f-a53d-71b2a64f3325-kube-api-access-csxjx\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb\" (UID: \"75319d6b-af9b-4c2f-a53d-71b2a64f3325\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"
Nov 01 06:33:10.244200 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:10.244161    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"
Nov 01 06:33:10.621842 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:10.621813    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"]
Nov 01 06:33:10.622182 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:33:10.622160    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75319d6b_af9b_4c2f_a53d_71b2a64f3325.slice/crio-dae8ada0e98cc0a9d0cc141404a1b70aa82f135395df46444a86545a38ae95c7 WatchSource:0}: Error finding container dae8ada0e98cc0a9d0cc141404a1b70aa82f135395df46444a86545a38ae95c7: Status 404 returned error can't find the container with id dae8ada0e98cc0a9d0cc141404a1b70aa82f135395df46444a86545a38ae95c7
Nov 01 06:33:11.228305 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:11.228256    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb" event={"ID":"75319d6b-af9b-4c2f-a53d-71b2a64f3325","Type":"ContainerStarted","Data":"72e3023d326aaa7932abfa30307b5ebd93c279f94f0a5212dfa8a73d0e7a69a5"}
Nov 01 06:33:11.228809 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:11.228392    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb" event={"ID":"75319d6b-af9b-4c2f-a53d-71b2a64f3325","Type":"ContainerStarted","Data":"dae8ada0e98cc0a9d0cc141404a1b70aa82f135395df46444a86545a38ae95c7"}
Nov 01 06:33:11.229663 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:11.229635    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"
Nov 01 06:33:11.349593 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:11.349541    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb" podStartSLOduration=2.349526651 podStartE2EDuration="2.349526651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:33:11.348447444 +0000 UTC m=+2828.559270953" watchObservedRunningTime="2026-11-01 06:33:11.349526651 +0000 UTC m=+2828.560350163"
Nov 01 06:33:14.676959 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:14.676921    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:14.873360 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:14.873298    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj" podUID="e98c4ffb-6a21-459d-be1b-0bde6489e0c1" containerName="payload-processing" containerID="cri-o://009aa7ec8853dd5bdac1828bc491ab34771daa018926c4b165e17b7aea07803a" gracePeriod=25
Nov 01 06:33:15.139998 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:15.139972    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"
Nov 01 06:33:15.230081 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:15.230051    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-routing-overlay-volume\") pod \"e98c4ffb-6a21-459d-be1b-0bde6489e0c1\" (UID: \"e98c4ffb-6a21-459d-be1b-0bde6489e0c1\") "
Nov 01 06:33:15.230249 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:15.230098    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-kube-api-access-5l79n\" (UniqueName: \"kubernetes.io/projected/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-kube-api-access-5l79n\") pod \"e98c4ffb-6a21-459d-be1b-0bde6489e0c1\" (UID: \"e98c4ffb-6a21-459d-be1b-0bde6489e0c1\") "
Nov 01 06:33:15.230249 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:15.230218    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-provider-credentials\") pod \"e98c4ffb-6a21-459d-be1b-0bde6489e0c1\" (UID: \"e98c4ffb-6a21-459d-be1b-0bde6489e0c1\") "
Nov 01 06:33:15.230361 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:15.230310    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-plugins-config-volume\") pod \"e98c4ffb-6a21-459d-be1b-0bde6489e0c1\" (UID: \"e98c4ffb-6a21-459d-be1b-0bde6489e0c1\") "
Nov 01 06:33:15.230419 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:15.230392    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-routing-overlay-volume" pod "e98c4ffb-6a21-459d-be1b-0bde6489e0c1" (UID: "e98c4ffb-6a21-459d-be1b-0bde6489e0c1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:33:15.230675 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:15.230656    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:15.230776 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:15.230729    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-plugins-config-volume" pod "e98c4ffb-6a21-459d-be1b-0bde6489e0c1" (UID: "e98c4ffb-6a21-459d-be1b-0bde6489e0c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:33:15.232336 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:15.232303    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-provider-credentials" pod "e98c4ffb-6a21-459d-be1b-0bde6489e0c1" (UID: "e98c4ffb-6a21-459d-be1b-0bde6489e0c1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:33:15.232431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:15.232380    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-kube-api-access-5l79n" pod "e98c4ffb-6a21-459d-be1b-0bde6489e0c1" (UID: "e98c4ffb-6a21-459d-be1b-0bde6489e0c1"). InnerVolumeSpecName "kube-api-access-5l79n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:33:15.254779 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:15.254707    5619 generic.go:356] "Generic (PLEG): container finished" podID="e98c4ffb-6a21-459d-be1b-0bde6489e0c1" containerID="009aa7ec8853dd5bdac1828bc491ab34771daa018926c4b165e17b7aea07803a" exitCode=0
Nov 01 06:33:15.254779 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:15.254768    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"
Nov 01 06:33:15.254960 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:15.254805    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj" event={"ID":"e98c4ffb-6a21-459d-be1b-0bde6489e0c1","Type":"ContainerDied","Data":"009aa7ec8853dd5bdac1828bc491ab34771daa018926c4b165e17b7aea07803a"}
Nov 01 06:33:15.254960 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:15.254905    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj" event={"ID":"e98c4ffb-6a21-459d-be1b-0bde6489e0c1","Type":"ContainerDied","Data":"9e18d0a54b631fde67252c7fd2f467d3957d09fd9a61a46a13de9d70e272bc30"}
Nov 01 06:33:15.254960 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:15.254941    5619 scope.go:122] "RemoveContainer" containerID="009aa7ec8853dd5bdac1828bc491ab34771daa018926c4b165e17b7aea07803a"
Nov 01 06:33:15.267935 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:15.267904    5619 scope.go:122] "RemoveContainer" containerID="009aa7ec8853dd5bdac1828bc491ab34771daa018926c4b165e17b7aea07803a"
Nov 01 06:33:15.268300 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:33:15.268270    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"009aa7ec8853dd5bdac1828bc491ab34771daa018926c4b165e17b7aea07803a\": container with ID starting with 009aa7ec8853dd5bdac1828bc491ab34771daa018926c4b165e17b7aea07803a not found: ID does not exist" containerID="009aa7ec8853dd5bdac1828bc491ab34771daa018926c4b165e17b7aea07803a"
Nov 01 06:33:15.268527 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:15.268471    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"009aa7ec8853dd5bdac1828bc491ab34771daa018926c4b165e17b7aea07803a"} err="failed to get container status \"009aa7ec8853dd5bdac1828bc491ab34771daa018926c4b165e17b7aea07803a\": rpc error: code = NotFound desc = could not find container \"009aa7ec8853dd5bdac1828bc491ab34771daa018926c4b165e17b7aea07803a\": container with ID starting with 009aa7ec8853dd5bdac1828bc491ab34771daa018926c4b165e17b7aea07803a not found: ID does not exist"
Nov 01 06:33:15.331811 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:15.331776    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:15.331811 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:15.331806    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l79n\" (UniqueName: \"kubernetes.io/projected/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-kube-api-access-5l79n\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:15.331811 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:15.331816    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e98c4ffb-6a21-459d-be1b-0bde6489e0c1-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:15.426713 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:15.426680    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"]
Nov 01 06:33:15.430535 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:15.430508    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949ngvdj"]
Nov 01 06:33:16.965196 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:16.965150    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e98c4ffb-6a21-459d-be1b-0bde6489e0c1" path="/var/lib/kubelet/pods/e98c4ffb-6a21-459d-be1b-0bde6489e0c1/volumes"
Nov 01 06:33:19.572279 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:19.572242    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:23.243044 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:23.243017    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"
Nov 01 06:33:23.414671 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:23.414640    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"]
Nov 01 06:33:24.461872 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:24.461842    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"]
Nov 01 06:33:24.462506 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:24.462489    5619 cpu_manager.go:414] "containerMap: removing container" podUID="e98c4ffb-6a21-459d-be1b-0bde6489e0c1" containerName="payload-processing"
Nov 01 06:33:24.462506 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:24.462507    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e98c4ffb-6a21-459d-be1b-0bde6489e0c1" containerName="payload-processing"
Nov 01 06:33:24.462641 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:24.462594    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="e98c4ffb-6a21-459d-be1b-0bde6489e0c1" containerName="payload-processing"
Nov 01 06:33:24.480588 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:24.480561    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"
Nov 01 06:33:24.526229 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:24.526196    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"]
Nov 01 06:33:24.533856 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:24.533823    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-3680c5-serving-cert\""
Nov 01 06:33:24.647986 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:24.647952    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-maas-api-tls\") pod \"maas-api-e2e-autores-3680c5-7d84756df5-spl9h\" (UID: \"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"
Nov 01 06:33:24.648176 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:24.648034    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-openshift-service-ca\") pod \"maas-api-e2e-autores-3680c5-7d84756df5-spl9h\" (UID: \"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"
Nov 01 06:33:24.648176 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:24.648093    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-metrics-certs\") pod \"maas-api-e2e-autores-3680c5-7d84756df5-spl9h\" (UID: \"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"
Nov 01 06:33:24.648474 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:24.648442    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsspm\" (UniqueName: \"kubernetes.io/projected/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-kube-api-access-qsspm\") pod \"maas-api-e2e-autores-3680c5-7d84756df5-spl9h\" (UID: \"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"
Nov 01 06:33:24.702026 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:24.701987    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:24.749308 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:24.749214    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-metrics-certs\") pod \"maas-api-e2e-autores-3680c5-7d84756df5-spl9h\" (UID: \"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"
Nov 01 06:33:24.749499 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:24.749390    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsspm\" (UniqueName: \"kubernetes.io/projected/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-kube-api-access-qsspm\") pod \"maas-api-e2e-autores-3680c5-7d84756df5-spl9h\" (UID: \"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"
Nov 01 06:33:24.749499 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:24.749484    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-maas-api-tls\") pod \"maas-api-e2e-autores-3680c5-7d84756df5-spl9h\" (UID: \"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"
Nov 01 06:33:24.749645 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:24.749526    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-openshift-service-ca\") pod \"maas-api-e2e-autores-3680c5-7d84756df5-spl9h\" (UID: \"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"
Nov 01 06:33:24.749645 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:33:24.749594    5619 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-serving-cert: secret "maas-api-e2e-autores-3680c5-serving-cert" not found
Nov 01 06:33:24.749770 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:33:24.749660    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-maas-api-tls podName:6580ffaf-1ef0-47e4-b1b7-aa722e3117f2 nodeName:}" failed. No retries permitted until 2026-11-01 06:33:25.249638825 +0000 UTC m=+2842.460462326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-maas-api-tls") pod "maas-api-e2e-autores-3680c5-7d84756df5-spl9h" (UID: "6580ffaf-1ef0-47e4-b1b7-aa722e3117f2") : secret "maas-api-e2e-autores-3680c5-serving-cert" not found
Nov 01 06:33:24.750291 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:24.750262    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-openshift-service-ca\") pod \"maas-api-e2e-autores-3680c5-7d84756df5-spl9h\" (UID: \"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"
Nov 01 06:33:24.756395 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:24.751943    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-metrics-certs\") pod \"maas-api-e2e-autores-3680c5-7d84756df5-spl9h\" (UID: \"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"
Nov 01 06:33:24.811830 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:24.811789    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsspm\" (UniqueName: \"kubernetes.io/projected/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-kube-api-access-qsspm\") pod \"maas-api-e2e-autores-3680c5-7d84756df5-spl9h\" (UID: \"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"
Nov 01 06:33:24.908295 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:24.908264    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:25.051729 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:25.051644    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:25.255821 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:25.255784    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-maas-api-tls\") pod \"maas-api-e2e-autores-3680c5-7d84756df5-spl9h\" (UID: \"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"
Nov 01 06:33:25.258169 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:25.258146    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-maas-api-tls\") pod \"maas-api-e2e-autores-3680c5-7d84756df5-spl9h\" (UID: \"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"
Nov 01 06:33:25.396200 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:25.396169    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"
Nov 01 06:33:25.778235 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:25.778206    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"]
Nov 01 06:33:25.779441 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:33:25.779411    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6580ffaf_1ef0_47e4_b1b7_aa722e3117f2.slice/crio-e5bdcf4302a150b65b5861a635b0294f82de2a6c2dea07db152686df916750d2 WatchSource:0}: Error finding container e5bdcf4302a150b65b5861a635b0294f82de2a6c2dea07db152686df916750d2: Status 404 returned error can't find the container with id e5bdcf4302a150b65b5861a635b0294f82de2a6c2dea07db152686df916750d2
Nov 01 06:33:25.878646 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:33:25.878616    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:33:26.325383 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:26.325345    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h" event={"ID":"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2","Type":"ContainerStarted","Data":"e5bdcf4302a150b65b5861a635b0294f82de2a6c2dea07db152686df916750d2"}
Nov 01 06:33:27.102842 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:27.102812    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:27.243042 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:27.243004    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:27.332227 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:27.332184    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h" event={"ID":"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2","Type":"ContainerStarted","Data":"5b525a79dacc17ad552ee03a9c483f8a2b8c6766ae755577c5dd217295532c10"}
Nov 01 06:33:27.332421 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:27.332347    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"
Nov 01 06:33:27.394654 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:27.394569    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:27.455951 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:27.455897    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h" podStartSLOduration=2.586778479 podStartE2EDuration="3.455884101s" totalImagesPullingTime="869.105622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:33:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:33:27.454060451 +0000 UTC m=+2844.664883959" watchObservedRunningTime="2026-11-01 06:33:27.455884101 +0000 UTC m=+2844.666707607"
Nov 01 06:33:27.833846 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:27.833769    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:28.415014 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:28.414961    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf" podUID="fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a" containerName="payload-processing" containerID="cri-o://ec1a9f518ffab195864711366b1319fcb44e39bda5ee45caed5c4298cd013cde" gracePeriod=25
Nov 01 06:33:28.683263 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:28.683201    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"
Nov 01 06:33:28.699549 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:28.699525    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-routing-overlay-volume\") pod \"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a\" (UID: \"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a\") "
Nov 01 06:33:28.699674 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:28.699554    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-kube-api-access-rzvbb\" (UniqueName: \"kubernetes.io/projected/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-kube-api-access-rzvbb\") pod \"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a\" (UID: \"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a\") "
Nov 01 06:33:28.699674 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:28.699593    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-provider-credentials\") pod \"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a\" (UID: \"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a\") "
Nov 01 06:33:28.699770 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:28.699707    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-plugins-config-volume\") pod \"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a\" (UID: \"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a\") "
Nov 01 06:33:28.699919 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:28.699889    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-routing-overlay-volume" pod "fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a" (UID: "fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:33:28.700116 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:28.700094    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-plugins-config-volume" pod "fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a" (UID: "fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:33:28.700367 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:28.700347    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:28.700453 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:28.700369    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:28.701862 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:28.701831    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-provider-credentials" pod "fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a" (UID: "fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:33:28.701962 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:28.701897    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-kube-api-access-rzvbb" pod "fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a" (UID: "fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a"). InnerVolumeSpecName "kube-api-access-rzvbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:33:28.801776 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:28.801742    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzvbb\" (UniqueName: \"kubernetes.io/projected/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-kube-api-access-rzvbb\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:28.801776 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:28.801773    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:29.345306 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:29.345274    5619 generic.go:356] "Generic (PLEG): container finished" podID="fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a" containerID="ec1a9f518ffab195864711366b1319fcb44e39bda5ee45caed5c4298cd013cde" exitCode=0
Nov 01 06:33:29.345533 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:29.345318    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"
Nov 01 06:33:29.345533 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:29.345369    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf" event={"ID":"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a","Type":"ContainerDied","Data":"ec1a9f518ffab195864711366b1319fcb44e39bda5ee45caed5c4298cd013cde"}
Nov 01 06:33:29.345533 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:29.345433    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf" event={"ID":"fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a","Type":"ContainerDied","Data":"bc62c1e205e5ebf7963c7e13ffce609f30f16de03daa8e410492dd4ffcd537c6"}
Nov 01 06:33:29.345533 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:29.345465    5619 scope.go:122] "RemoveContainer" containerID="ec1a9f518ffab195864711366b1319fcb44e39bda5ee45caed5c4298cd013cde"
Nov 01 06:33:29.354641 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:29.354619    5619 scope.go:122] "RemoveContainer" containerID="ec1a9f518ffab195864711366b1319fcb44e39bda5ee45caed5c4298cd013cde"
Nov 01 06:33:29.354933 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:33:29.354912    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec1a9f518ffab195864711366b1319fcb44e39bda5ee45caed5c4298cd013cde\": container with ID starting with ec1a9f518ffab195864711366b1319fcb44e39bda5ee45caed5c4298cd013cde not found: ID does not exist" containerID="ec1a9f518ffab195864711366b1319fcb44e39bda5ee45caed5c4298cd013cde"
Nov 01 06:33:29.354990 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:29.354941    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1a9f518ffab195864711366b1319fcb44e39bda5ee45caed5c4298cd013cde"} err="failed to get container status \"ec1a9f518ffab195864711366b1319fcb44e39bda5ee45caed5c4298cd013cde\": rpc error: code = NotFound desc = could not find container \"ec1a9f518ffab195864711366b1319fcb44e39bda5ee45caed5c4298cd013cde\": container with ID starting with ec1a9f518ffab195864711366b1319fcb44e39bda5ee45caed5c4298cd013cde not found: ID does not exist"
Nov 01 06:33:29.515080 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:29.515053    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"]
Nov 01 06:33:29.565752 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:29.565722    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6999bjdnf"]
Nov 01 06:33:29.690486 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:29.690395    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:30.888264 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:30.888228    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:30.963058 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:30.963025    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a" path="/var/lib/kubelet/pods/fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a/volumes"
Nov 01 06:33:32.040170 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.040137    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"]
Nov 01 06:33:32.040980 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.040956    5619 cpu_manager.go:414] "containerMap: removing container" podUID="fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a" containerName="payload-processing"
Nov 01 06:33:32.041083 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.040981    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a" containerName="payload-processing"
Nov 01 06:33:32.041152 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.041139    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdd031b4-43b6-4cc6-9991-9f1c92f6fb7a" containerName="payload-processing"
Nov 01 06:33:32.045703 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.045682    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"
Nov 01 06:33:32.100911 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.100833    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9f8bf4cbae3d\""
Nov 01 06:33:32.101107 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.101070    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 06:33:32.101226 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.101135    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 06:33:32.136105 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.136075    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03c227a0-506e-446e-a0ef-b9650a02b21f-routing-overlay-volume\") pod \"payload-processing-external-model-76b56d9557-kpktp\" (UID: \"03c227a0-506e-446e-a0ef-b9650a02b21f\") " pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"
Nov 01 06:33:32.136244 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.136125    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-provider-credentials\") pod \"payload-processing-external-model-76b56d9557-kpktp\" (UID: \"03c227a0-506e-446e-a0ef-b9650a02b21f\") " pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"
Nov 01 06:33:32.136244 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.136226    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4z87\" (UniqueName: \"kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-kube-api-access-q4z87\") pod \"payload-processing-external-model-76b56d9557-kpktp\" (UID: \"03c227a0-506e-446e-a0ef-b9650a02b21f\") " pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"
Nov 01 06:33:32.136341 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.136296    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c227a0-506e-446e-a0ef-b9650a02b21f-plugins-config-volume\") pod \"payload-processing-external-model-76b56d9557-kpktp\" (UID: \"03c227a0-506e-446e-a0ef-b9650a02b21f\") " pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"
Nov 01 06:33:32.151674 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.151644    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 06:33:32.151872 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.151855    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 06:33:32.152074 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.152020    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-qvhf2\""
Nov 01 06:33:32.152150 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.152129    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1b1e343b358e\""
Nov 01 06:33:32.154913 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.154891    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"]
Nov 01 06:33:32.162607 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.162586    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm"]
Nov 01 06:33:32.237311 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.237281    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03c227a0-506e-446e-a0ef-b9650a02b21f-routing-overlay-volume\") pod \"payload-processing-external-model-76b56d9557-kpktp\" (UID: \"03c227a0-506e-446e-a0ef-b9650a02b21f\") " pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"
Nov 01 06:33:32.237483 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.237426    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-provider-credentials\") pod \"payload-processing-external-model-76b56d9557-kpktp\" (UID: \"03c227a0-506e-446e-a0ef-b9650a02b21f\") " pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"
Nov 01 06:33:32.237483 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.237475    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4z87\" (UniqueName: \"kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-kube-api-access-q4z87\") pod \"payload-processing-external-model-76b56d9557-kpktp\" (UID: \"03c227a0-506e-446e-a0ef-b9650a02b21f\") " pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"
Nov 01 06:33:32.237598 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.237548    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c227a0-506e-446e-a0ef-b9650a02b21f-plugins-config-volume\") pod \"payload-processing-external-model-76b56d9557-kpktp\" (UID: \"03c227a0-506e-446e-a0ef-b9650a02b21f\") " pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"
Nov 01 06:33:32.237974 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.237951    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03c227a0-506e-446e-a0ef-b9650a02b21f-routing-overlay-volume\") pod \"payload-processing-external-model-76b56d9557-kpktp\" (UID: \"03c227a0-506e-446e-a0ef-b9650a02b21f\") " pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"
Nov 01 06:33:32.238222 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.238203    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c227a0-506e-446e-a0ef-b9650a02b21f-plugins-config-volume\") pod \"payload-processing-external-model-76b56d9557-kpktp\" (UID: \"03c227a0-506e-446e-a0ef-b9650a02b21f\") " pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"
Nov 01 06:33:32.239962 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.239938    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-provider-credentials\") pod \"payload-processing-external-model-76b56d9557-kpktp\" (UID: \"03c227a0-506e-446e-a0ef-b9650a02b21f\") " pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"
Nov 01 06:33:32.271587 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.269200    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk"]
Nov 01 06:33:32.299368 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.299285    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4z87\" (UniqueName: \"kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-kube-api-access-q4z87\") pod \"payload-processing-external-model-76b56d9557-kpktp\" (UID: \"03c227a0-506e-446e-a0ef-b9650a02b21f\") " pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"
Nov 01 06:33:32.358710 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.358677    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"
Nov 01 06:33:32.516042 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.516002    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:32.704025 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.703011    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:32.746585 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:32.744458    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"]
Nov 01 06:33:32.753365 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:33:32.753316    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c227a0_506e_446e_a0ef_b9650a02b21f.slice/crio-4e0d9d49fbd0929256ccbb2f0351c179eb8fa819f2259525ebdc22241e0ff851 WatchSource:0}: Error finding container 4e0d9d49fbd0929256ccbb2f0351c179eb8fa819f2259525ebdc22241e0ff851: Status 404 returned error can't find the container with id 4e0d9d49fbd0929256ccbb2f0351c179eb8fa819f2259525ebdc22241e0ff851
Nov 01 06:33:33.344110 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:33.344085    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"
Nov 01 06:33:33.372998 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:33.372949    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp" event={"ID":"03c227a0-506e-446e-a0ef-b9650a02b21f","Type":"ContainerStarted","Data":"8c5587441fff843eaac1bdcd2f6d1159f25d977c849ca0beec4ec5a2ee00a4be"}
Nov 01 06:33:33.373160 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:33.373040    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"
Nov 01 06:33:33.373160 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:33.373058    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp" event={"ID":"03c227a0-506e-446e-a0ef-b9650a02b21f","Type":"ContainerStarted","Data":"4e0d9d49fbd0929256ccbb2f0351c179eb8fa819f2259525ebdc22241e0ff851"}
Nov 01 06:33:33.583506 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:33.583452    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp" podStartSLOduration=2.583437902 podStartE2EDuration="2.583437902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:33:33.582098367 +0000 UTC m=+2850.792921878" watchObservedRunningTime="2026-11-01 06:33:33.583437902 +0000 UTC m=+2850.794261410"
Nov 01 06:33:34.483360 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:34.483286    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:34.625373 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:34.625340    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:37.163618 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:37.163559    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm" podUID="0f2b2a1a-9247-47c1-bf89-65210876c41f" containerName="payload-processing" containerID="cri-o://afa4a23afcb4cb57ef867d9c8995d2c0bfe82a7c2c12e8b71d1f77a56058877b" gracePeriod=25
Nov 01 06:33:37.270535 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:37.270497    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk" podUID="0dd5afce-97a4-4a99-b303-3e542ef0ddba" containerName="payload-pre-processing" containerID="cri-o://8d1dbf22633101406f7f3bf6d767081aa359032401e316f88a48e9c20a0faff1" gracePeriod=25
Nov 01 06:33:39.265261 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:39.265219    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:44.172415 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:44.172359    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:44.382448 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:44.382419    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"
Nov 01 06:33:46.508257 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:46.508213    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm" podUID="0f2b2a1a-9247-47c1-bf89-65210876c41f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:33:46.508695 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:46.508294    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk" podUID="0dd5afce-97a4-4a99-b303-3e542ef0ddba" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:33:48.964509 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:48.964472    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:53.682595 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:53.682559    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:55.940729 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:33:55.940686    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:33:56.528509 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:56.528457    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm" podUID="0f2b2a1a-9247-47c1-bf89-65210876c41f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:33:56.529251 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:56.529220    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk" podUID="0dd5afce-97a4-4a99-b303-3e542ef0ddba" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:33:57.548517 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:57.548488    5619 generic.go:356] "Generic (PLEG): container finished" podID="0f2b2a1a-9247-47c1-bf89-65210876c41f" containerID="afa4a23afcb4cb57ef867d9c8995d2c0bfe82a7c2c12e8b71d1f77a56058877b" exitCode=0
Nov 01 06:33:57.548893 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:57.548565    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm" event={"ID":"0f2b2a1a-9247-47c1-bf89-65210876c41f","Type":"ContainerDied","Data":"afa4a23afcb4cb57ef867d9c8995d2c0bfe82a7c2c12e8b71d1f77a56058877b"}
Nov 01 06:33:57.548893 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:57.548603    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm" event={"ID":"0f2b2a1a-9247-47c1-bf89-65210876c41f","Type":"ContainerDied","Data":"c4bf272d7d6745b21987652d0ae855d0e8a9da5dfdbcb2025a0fe7b74454cfee"}
Nov 01 06:33:57.548893 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:57.548621    5619 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4bf272d7d6745b21987652d0ae855d0e8a9da5dfdbcb2025a0fe7b74454cfee"
Nov 01 06:33:57.550479 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:57.550454    5619 generic.go:356] "Generic (PLEG): container finished" podID="0dd5afce-97a4-4a99-b303-3e542ef0ddba" containerID="8d1dbf22633101406f7f3bf6d767081aa359032401e316f88a48e9c20a0faff1" exitCode=0
Nov 01 06:33:57.550592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:57.550530    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk" event={"ID":"0dd5afce-97a4-4a99-b303-3e542ef0ddba","Type":"ContainerDied","Data":"8d1dbf22633101406f7f3bf6d767081aa359032401e316f88a48e9c20a0faff1"}
Nov 01 06:33:57.566367 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:57.566346    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm"
Nov 01 06:33:57.590093 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:57.590073    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk"
Nov 01 06:33:57.628169 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:57.628143    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd5afce-97a4-4a99-b303-3e542ef0ddba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd5afce-97a4-4a99-b303-3e542ef0ddba-plugins-config-volume\") pod \"0dd5afce-97a4-4a99-b303-3e542ef0ddba\" (UID: \"0dd5afce-97a4-4a99-b303-3e542ef0ddba\") "
Nov 01 06:33:57.628342 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:57.628218    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd5afce-97a4-4a99-b303-3e542ef0ddba-kube-api-access-bpcgp\" (UniqueName: \"kubernetes.io/projected/0dd5afce-97a4-4a99-b303-3e542ef0ddba-kube-api-access-bpcgp\") pod \"0dd5afce-97a4-4a99-b303-3e542ef0ddba\" (UID: \"0dd5afce-97a4-4a99-b303-3e542ef0ddba\") "
Nov 01 06:33:57.628342 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:57.628277    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f2b2a1a-9247-47c1-bf89-65210876c41f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f2b2a1a-9247-47c1-bf89-65210876c41f-plugins-config-volume\") pod \"0f2b2a1a-9247-47c1-bf89-65210876c41f\" (UID: \"0f2b2a1a-9247-47c1-bf89-65210876c41f\") "
Nov 01 06:33:57.628475 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:57.628361    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2b2a1a-9247-47c1-bf89-65210876c41f-kube-api-access-vngm5\" (UniqueName: \"kubernetes.io/projected/0f2b2a1a-9247-47c1-bf89-65210876c41f-kube-api-access-vngm5\") pod \"0f2b2a1a-9247-47c1-bf89-65210876c41f\" (UID: \"0f2b2a1a-9247-47c1-bf89-65210876c41f\") "
Nov 01 06:33:57.628550 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:57.628519    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd5afce-97a4-4a99-b303-3e542ef0ddba-plugins-config-volume" pod "0dd5afce-97a4-4a99-b303-3e542ef0ddba" (UID: "0dd5afce-97a4-4a99-b303-3e542ef0ddba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:33:57.628796 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:57.628771    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2b2a1a-9247-47c1-bf89-65210876c41f-plugins-config-volume" pod "0f2b2a1a-9247-47c1-bf89-65210876c41f" (UID: "0f2b2a1a-9247-47c1-bf89-65210876c41f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:33:57.629003 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:57.628985    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f2b2a1a-9247-47c1-bf89-65210876c41f-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:57.629069 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:57.629010    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd5afce-97a4-4a99-b303-3e542ef0ddba-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:57.630420 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:57.630401    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2b2a1a-9247-47c1-bf89-65210876c41f-kube-api-access-vngm5" pod "0f2b2a1a-9247-47c1-bf89-65210876c41f" (UID: "0f2b2a1a-9247-47c1-bf89-65210876c41f"). InnerVolumeSpecName "kube-api-access-vngm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:33:57.630473 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:57.630427    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd5afce-97a4-4a99-b303-3e542ef0ddba-kube-api-access-bpcgp" pod "0dd5afce-97a4-4a99-b303-3e542ef0ddba" (UID: "0dd5afce-97a4-4a99-b303-3e542ef0ddba"). InnerVolumeSpecName "kube-api-access-bpcgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:33:57.729917 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:57.729888    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpcgp\" (UniqueName: \"kubernetes.io/projected/0dd5afce-97a4-4a99-b303-3e542ef0ddba-kube-api-access-bpcgp\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:57.729917 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:57.729915    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vngm5\" (UniqueName: \"kubernetes.io/projected/0f2b2a1a-9247-47c1-bf89-65210876c41f-kube-api-access-vngm5\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:33:58.560567 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:58.560315    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk"
Nov 01 06:33:58.560567 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:58.560343    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk" event={"ID":"0dd5afce-97a4-4a99-b303-3e542ef0ddba","Type":"ContainerDied","Data":"07b1090600579ed673070b80f63821518c98c1208d7e6efcdf149b390e9a408b"}
Nov 01 06:33:58.560567 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:58.560494    5619 scope.go:122] "RemoveContainer" containerID="8d1dbf22633101406f7f3bf6d767081aa359032401e316f88a48e9c20a0faff1"
Nov 01 06:33:58.561136 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:58.560679    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm"
Nov 01 06:33:58.574896 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:58.574868    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:33:58.728498 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:58.728463    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm"]
Nov 01 06:33:58.780035 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:58.779998    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c0662f25-66567445f5-2w4wm"]
Nov 01 06:33:58.845043 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:58.845015    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk"]
Nov 01 06:33:58.848827 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:58.848803    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c0662f25-7c6dbc9655-rfqzk"]
Nov 01 06:33:58.963758 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:58.963729    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd5afce-97a4-4a99-b303-3e542ef0ddba" path="/var/lib/kubelet/pods/0dd5afce-97a4-4a99-b303-3e542ef0ddba/volumes"
Nov 01 06:33:58.964207 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:33:58.964191    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f2b2a1a-9247-47c1-bf89-65210876c41f" path="/var/lib/kubelet/pods/0f2b2a1a-9247-47c1-bf89-65210876c41f/volumes"
Nov 01 06:34:03.359806 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:03.359758    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:08.097719 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:08.097678    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:08.259553 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:08.259521    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:08.643981 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:08.643944    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:11.258242 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:11.258139    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:12.409179 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.409147    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"]
Nov 01 06:34:12.409789 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.409773    5619 cpu_manager.go:414] "containerMap: removing container" podUID="0dd5afce-97a4-4a99-b303-3e542ef0ddba" containerName="payload-pre-processing"
Nov 01 06:34:12.409842 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.409791    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd5afce-97a4-4a99-b303-3e542ef0ddba" containerName="payload-pre-processing"
Nov 01 06:34:12.409842 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.409822    5619 cpu_manager.go:414] "containerMap: removing container" podUID="0f2b2a1a-9247-47c1-bf89-65210876c41f" containerName="payload-processing"
Nov 01 06:34:12.409842 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.409827    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2b2a1a-9247-47c1-bf89-65210876c41f" containerName="payload-processing"
Nov 01 06:34:12.409948 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.409905    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd5afce-97a4-4a99-b303-3e542ef0ddba" containerName="payload-pre-processing"
Nov 01 06:34:12.409948 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.409918    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2b2a1a-9247-47c1-bf89-65210876c41f" containerName="payload-processing"
Nov 01 06:34:12.415213 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.415193    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"
Nov 01 06:34:12.471245 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.471218    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"]
Nov 01 06:34:12.512892 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.512856    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l\" (UID: \"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"
Nov 01 06:34:12.513063 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.512980    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l\" (UID: \"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"
Nov 01 06:34:12.513162 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.513112    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddxts\" (UniqueName: \"kubernetes.io/projected/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-kube-api-access-ddxts\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l\" (UID: \"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"
Nov 01 06:34:12.513214 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.513160    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l\" (UID: \"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"
Nov 01 06:34:12.518978 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.518944    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cfe368\"/\"e2e-cmp-legacy-9dbd2e-provider-cred\""
Nov 01 06:34:12.518978 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.518944    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cfe368\"/\"openshift-service-ca.crt\""
Nov 01 06:34:12.518978 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.518974    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cfe368\"/\"kube-root-ca.crt\""
Nov 01 06:34:12.519213 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.519042    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cfe368\"/\"payload-processing-external-model-plugins-e2e-extmod-a-cfe368\""
Nov 01 06:34:12.519213 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.519048    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cfe368\"/\"payload-processing-external-model-e2e--6d943d76-dockercfg-6l4qj\""
Nov 01 06:34:12.519213 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.519109    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cfe368\"/\"routing-overlay\""
Nov 01 06:34:12.614103 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.614070    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l\" (UID: \"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"
Nov 01 06:34:12.614296 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.614193    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddxts\" (UniqueName: \"kubernetes.io/projected/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-kube-api-access-ddxts\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l\" (UID: \"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"
Nov 01 06:34:12.614296 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.614259    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l\" (UID: \"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"
Nov 01 06:34:12.614450 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.614373    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l\" (UID: \"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"
Nov 01 06:34:12.614952 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.614927    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l\" (UID: \"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"
Nov 01 06:34:12.615024 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.615010    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l\" (UID: \"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"
Nov 01 06:34:12.616780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.616754    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l\" (UID: \"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"
Nov 01 06:34:12.676428 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.676367    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddxts\" (UniqueName: \"kubernetes.io/projected/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-kube-api-access-ddxts\") pod \"payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l\" (UID: \"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1\") " pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"
Nov 01 06:34:12.729073 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.729039    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"
Nov 01 06:34:12.982694 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:12.982608    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:13.105602 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:13.105573    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"]
Nov 01 06:34:13.106382 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:34:13.106355    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0a06c3_710b_4eba_95d1_e6fd2b70f1c1.slice/crio-39ac297d84531419588e5e7dcff779cadf6bd5774f9b009c466462f82c2c6857 WatchSource:0}: Error finding container 39ac297d84531419588e5e7dcff779cadf6bd5774f9b009c466462f82c2c6857: Status 404 returned error can't find the container with id 39ac297d84531419588e5e7dcff779cadf6bd5774f9b009c466462f82c2c6857
Nov 01 06:34:13.657866 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:13.657832    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l" event={"ID":"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1","Type":"ContainerStarted","Data":"0c0763779f13a7eab27a60d9d533ac2bc6a7b0df8dac70100620eefa08e474b2"}
Nov 01 06:34:13.658252 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:13.657887    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l" event={"ID":"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1","Type":"ContainerStarted","Data":"39ac297d84531419588e5e7dcff779cadf6bd5774f9b009c466462f82c2c6857"}
Nov 01 06:34:13.658252 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:13.657950    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"
Nov 01 06:34:13.777816 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:13.777762    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l" podStartSLOduration=1.777746553 podStartE2EDuration="1.777746553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:34:13.776721399 +0000 UTC m=+2890.987544908" watchObservedRunningTime="2026-11-01 06:34:13.777746553 +0000 UTC m=+2890.988570059"
Nov 01 06:34:17.766667 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:17.766630    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:22.004347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:22.003439    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-76s2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:34:22.004347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:22.003511    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k" podUID="69cd086b-554c-49c9-a471-831d40610ceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:34:22.143556 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:22.143516    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-8c4ww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 06:34:22.143884 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:22.143836    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww" podUID="bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 06:34:22.867727 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:22.867689    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:24.666166 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:24.666124    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"
Nov 01 06:34:25.886027 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:34:25.885992    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:34:27.657218 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:27.657182    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"]
Nov 01 06:34:27.661541 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:27.661525    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"
Nov 01 06:34:27.689949 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:27.689906    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:27.742229 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:27.742196    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"]
Nov 01 06:34:27.821597 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:27.821558    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c777bdea-6aeb-48cf-80c7-97f263013624-plugins-config-volume\") pod \"payload-processing-external-model-5fbc98587d-qvn7c\" (UID: \"c777bdea-6aeb-48cf-80c7-97f263013624\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"
Nov 01 06:34:27.821774 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:27.821621    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c777bdea-6aeb-48cf-80c7-97f263013624-routing-overlay-volume\") pod \"payload-processing-external-model-5fbc98587d-qvn7c\" (UID: \"c777bdea-6aeb-48cf-80c7-97f263013624\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"
Nov 01 06:34:27.821774 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:27.821729    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vjtp\" (UniqueName: \"kubernetes.io/projected/c777bdea-6aeb-48cf-80c7-97f263013624-kube-api-access-4vjtp\") pod \"payload-processing-external-model-5fbc98587d-qvn7c\" (UID: \"c777bdea-6aeb-48cf-80c7-97f263013624\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"
Nov 01 06:34:27.821774 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:27.821765    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c777bdea-6aeb-48cf-80c7-97f263013624-provider-credentials\") pod \"payload-processing-external-model-5fbc98587d-qvn7c\" (UID: \"c777bdea-6aeb-48cf-80c7-97f263013624\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"
Nov 01 06:34:27.922897 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:27.922814    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c777bdea-6aeb-48cf-80c7-97f263013624-plugins-config-volume\") pod \"payload-processing-external-model-5fbc98587d-qvn7c\" (UID: \"c777bdea-6aeb-48cf-80c7-97f263013624\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"
Nov 01 06:34:27.922897 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:27.922874    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c777bdea-6aeb-48cf-80c7-97f263013624-routing-overlay-volume\") pod \"payload-processing-external-model-5fbc98587d-qvn7c\" (UID: \"c777bdea-6aeb-48cf-80c7-97f263013624\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"
Nov 01 06:34:27.923152 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:27.922920    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vjtp\" (UniqueName: \"kubernetes.io/projected/c777bdea-6aeb-48cf-80c7-97f263013624-kube-api-access-4vjtp\") pod \"payload-processing-external-model-5fbc98587d-qvn7c\" (UID: \"c777bdea-6aeb-48cf-80c7-97f263013624\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"
Nov 01 06:34:27.923152 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:27.922944    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c777bdea-6aeb-48cf-80c7-97f263013624-provider-credentials\") pod \"payload-processing-external-model-5fbc98587d-qvn7c\" (UID: \"c777bdea-6aeb-48cf-80c7-97f263013624\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"
Nov 01 06:34:27.923712 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:27.923686    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c777bdea-6aeb-48cf-80c7-97f263013624-routing-overlay-volume\") pod \"payload-processing-external-model-5fbc98587d-qvn7c\" (UID: \"c777bdea-6aeb-48cf-80c7-97f263013624\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"
Nov 01 06:34:27.923814 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:27.923686    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c777bdea-6aeb-48cf-80c7-97f263013624-plugins-config-volume\") pod \"payload-processing-external-model-5fbc98587d-qvn7c\" (UID: \"c777bdea-6aeb-48cf-80c7-97f263013624\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"
Nov 01 06:34:27.925478 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:27.925455    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c777bdea-6aeb-48cf-80c7-97f263013624-provider-credentials\") pod \"payload-processing-external-model-5fbc98587d-qvn7c\" (UID: \"c777bdea-6aeb-48cf-80c7-97f263013624\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"
Nov 01 06:34:28.016171 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:28.016144    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vjtp\" (UniqueName: \"kubernetes.io/projected/c777bdea-6aeb-48cf-80c7-97f263013624-kube-api-access-4vjtp\") pod \"payload-processing-external-model-5fbc98587d-qvn7c\" (UID: \"c777bdea-6aeb-48cf-80c7-97f263013624\") " pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"
Nov 01 06:34:28.203338 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:28.203230    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:28.275671 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:28.275640    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"
Nov 01 06:34:28.656297 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:28.656273    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"]
Nov 01 06:34:28.657316 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:34:28.657292    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc777bdea_6aeb_48cf_80c7_97f263013624.slice/crio-62ae7a308b5b740e3b5d5490d32cc4ca6975cac19422a745607daf8e33abe197 WatchSource:0}: Error finding container 62ae7a308b5b740e3b5d5490d32cc4ca6975cac19422a745607daf8e33abe197: Status 404 returned error can't find the container with id 62ae7a308b5b740e3b5d5490d32cc4ca6975cac19422a745607daf8e33abe197
Nov 01 06:34:28.780389 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:28.780353    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c" event={"ID":"c777bdea-6aeb-48cf-80c7-97f263013624","Type":"ContainerStarted","Data":"a7f486ed0cf861320ee1784afdfcfcedbe18e4345c283cb459fb03c448405a70"}
Nov 01 06:34:28.780548 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:28.780412    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c" event={"ID":"c777bdea-6aeb-48cf-80c7-97f263013624","Type":"ContainerStarted","Data":"62ae7a308b5b740e3b5d5490d32cc4ca6975cac19422a745607daf8e33abe197"}
Nov 01 06:34:28.780548 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:28.780438    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"
Nov 01 06:34:28.899908 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:28.899853    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c" podStartSLOduration=1.899837832 podStartE2EDuration="1.899837832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:34:28.897958808 +0000 UTC m=+2906.108782318" watchObservedRunningTime="2026-11-01 06:34:28.899837832 +0000 UTC m=+2906.110661340"
Nov 01 06:34:32.492033 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:32.492000    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:32.872619 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:32.872579    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:37.466429 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:37.466392    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:39.789993 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:39.789946    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c" podUID="c777bdea-6aeb-48cf-80c7-97f263013624" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:34:42.170820 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:42.170779    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:46.837339 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:46.837300    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:47.053951 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:47.053917    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:49.735630 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:49.735594    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:49.789060 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:49.789003    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c" podUID="c777bdea-6aeb-48cf-80c7-97f263013624" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:34:51.854513 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:51.854476    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:54.396217 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:54.396180    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:54.591156 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:54.591106    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:54.735052 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:54.734660    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:55.080197 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:34:55.080112    5619 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9f8bf4cbae3d: secret "agc-cred-9f8bf4cbae3d" not found
Nov 01 06:34:55.080197 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:34:55.080139    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-76b56d9557-kpktp: secret "agc-cred-9f8bf4cbae3d" not found
Nov 01 06:34:55.080532 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:34:55.080220    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-provider-credentials podName:03c227a0-506e-446e-a0ef-b9650a02b21f nodeName:}" failed. No retries permitted until 2026-11-01 06:34:55.580203133 +0000 UTC m=+2932.791026621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-provider-credentials") pod "payload-processing-external-model-76b56d9557-kpktp" (UID: "03c227a0-506e-446e-a0ef-b9650a02b21f") : secret "agc-cred-9f8bf4cbae3d" not found
Nov 01 06:34:55.587002 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:34:55.586971    5619 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9f8bf4cbae3d: secret "agc-cred-9f8bf4cbae3d" not found
Nov 01 06:34:55.587002 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:34:55.586997    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-76b56d9557-kpktp: secret "agc-cred-9f8bf4cbae3d" not found
Nov 01 06:34:55.587451 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:34:55.587062    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-provider-credentials podName:03c227a0-506e-446e-a0ef-b9650a02b21f nodeName:}" failed. No retries permitted until 2026-11-01 06:34:56.587045214 +0000 UTC m=+2933.797868701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-provider-credentials") pod "payload-processing-external-model-76b56d9557-kpktp" (UID: "03c227a0-506e-446e-a0ef-b9650a02b21f") : secret "agc-cred-9f8bf4cbae3d" not found
Nov 01 06:34:55.881102 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:34:55.881060    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:34:56.598656 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:34:56.598616    5619 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9f8bf4cbae3d: secret "agc-cred-9f8bf4cbae3d" not found
Nov 01 06:34:56.598656 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:34:56.598645    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-76b56d9557-kpktp: secret "agc-cred-9f8bf4cbae3d" not found
Nov 01 06:34:56.599180 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:34:56.598716    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-provider-credentials podName:03c227a0-506e-446e-a0ef-b9650a02b21f nodeName:}" failed. No retries permitted until 2026-11-01 06:34:58.598698062 +0000 UTC m=+2935.809521564 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-provider-credentials") pod "payload-processing-external-model-76b56d9557-kpktp" (UID: "03c227a0-506e-446e-a0ef-b9650a02b21f") : secret "agc-cred-9f8bf4cbae3d" not found
Nov 01 06:34:56.660744 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:56.660696    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:34:57.643612 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:57.643575    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"]
Nov 01 06:34:57.644959 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:34:57.644929    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c" podUID="c777bdea-6aeb-48cf-80c7-97f263013624" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:34:58.626313 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:34:58.626279    5619 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9f8bf4cbae3d: secret "agc-cred-9f8bf4cbae3d" not found
Nov 01 06:34:58.626313 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:34:58.626306    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-76b56d9557-kpktp: secret "agc-cred-9f8bf4cbae3d" not found
Nov 01 06:34:58.626572 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:34:58.626394    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-provider-credentials podName:03c227a0-506e-446e-a0ef-b9650a02b21f nodeName:}" failed. No retries permitted until 2026-11-01 06:35:02.626377579 +0000 UTC m=+2939.837201067 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-provider-credentials") pod "payload-processing-external-model-76b56d9557-kpktp" (UID: "03c227a0-506e-446e-a0ef-b9650a02b21f") : secret "agc-cred-9f8bf4cbae3d" not found
Nov 01 06:35:01.471891 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:01.471847    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:02.203752 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:02.203714    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:02.644690 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:02.644644    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c" podUID="c777bdea-6aeb-48cf-80c7-97f263013624" containerName="payload-processing" containerID="cri-o://a7f486ed0cf861320ee1784afdfcfcedbe18e4345c283cb459fb03c448405a70" gracePeriod=25
Nov 01 06:35:02.684232 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:35:02.684209    5619 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9f8bf4cbae3d: secret "agc-cred-9f8bf4cbae3d" not found
Nov 01 06:35:02.684232 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:35:02.684230    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-76b56d9557-kpktp: secret "agc-cred-9f8bf4cbae3d" not found
Nov 01 06:35:02.684410 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:35:02.684286    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-provider-credentials podName:03c227a0-506e-446e-a0ef-b9650a02b21f nodeName:}" failed. No retries permitted until 2026-11-01 06:35:10.684271129 +0000 UTC m=+2947.895094616 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-provider-credentials") pod "payload-processing-external-model-76b56d9557-kpktp" (UID: "03c227a0-506e-446e-a0ef-b9650a02b21f") : secret "agc-cred-9f8bf4cbae3d" not found
Nov 01 06:35:02.902591 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:02.902534    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"
Nov 01 06:35:02.988393 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:02.988363    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c777bdea-6aeb-48cf-80c7-97f263013624-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c777bdea-6aeb-48cf-80c7-97f263013624-plugins-config-volume\") pod \"c777bdea-6aeb-48cf-80c7-97f263013624\" (UID: \"c777bdea-6aeb-48cf-80c7-97f263013624\") "
Nov 01 06:35:02.988564 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:02.988397    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c777bdea-6aeb-48cf-80c7-97f263013624-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c777bdea-6aeb-48cf-80c7-97f263013624-provider-credentials\") pod \"c777bdea-6aeb-48cf-80c7-97f263013624\" (UID: \"c777bdea-6aeb-48cf-80c7-97f263013624\") "
Nov 01 06:35:02.988564 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:02.988420    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c777bdea-6aeb-48cf-80c7-97f263013624-kube-api-access-4vjtp\" (UniqueName: \"kubernetes.io/projected/c777bdea-6aeb-48cf-80c7-97f263013624-kube-api-access-4vjtp\") pod \"c777bdea-6aeb-48cf-80c7-97f263013624\" (UID: \"c777bdea-6aeb-48cf-80c7-97f263013624\") "
Nov 01 06:35:02.988564 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:02.988468    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c777bdea-6aeb-48cf-80c7-97f263013624-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c777bdea-6aeb-48cf-80c7-97f263013624-routing-overlay-volume\") pod \"c777bdea-6aeb-48cf-80c7-97f263013624\" (UID: \"c777bdea-6aeb-48cf-80c7-97f263013624\") "
Nov 01 06:35:02.988836 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:02.988813    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c777bdea-6aeb-48cf-80c7-97f263013624-plugins-config-volume" pod "c777bdea-6aeb-48cf-80c7-97f263013624" (UID: "c777bdea-6aeb-48cf-80c7-97f263013624"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:35:02.988911 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:02.988876    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c777bdea-6aeb-48cf-80c7-97f263013624-routing-overlay-volume" pod "c777bdea-6aeb-48cf-80c7-97f263013624" (UID: "c777bdea-6aeb-48cf-80c7-97f263013624"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:35:02.988982 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:02.988966    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c777bdea-6aeb-48cf-80c7-97f263013624-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:02.990532 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:02.990510    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c777bdea-6aeb-48cf-80c7-97f263013624-provider-credentials" pod "c777bdea-6aeb-48cf-80c7-97f263013624" (UID: "c777bdea-6aeb-48cf-80c7-97f263013624"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:35:02.990663 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:02.990621    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c777bdea-6aeb-48cf-80c7-97f263013624-kube-api-access-4vjtp" pod "c777bdea-6aeb-48cf-80c7-97f263013624" (UID: "c777bdea-6aeb-48cf-80c7-97f263013624"). InnerVolumeSpecName "kube-api-access-4vjtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:35:03.039164 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:03.039135    5619 generic.go:356] "Generic (PLEG): container finished" podID="c777bdea-6aeb-48cf-80c7-97f263013624" containerID="a7f486ed0cf861320ee1784afdfcfcedbe18e4345c283cb459fb03c448405a70" exitCode=0
Nov 01 06:35:03.039310 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:03.039176    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c" event={"ID":"c777bdea-6aeb-48cf-80c7-97f263013624","Type":"ContainerDied","Data":"a7f486ed0cf861320ee1784afdfcfcedbe18e4345c283cb459fb03c448405a70"}
Nov 01 06:35:03.039310 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:03.039195    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"
Nov 01 06:35:03.039310 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:03.039243    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c" event={"ID":"c777bdea-6aeb-48cf-80c7-97f263013624","Type":"ContainerDied","Data":"62ae7a308b5b740e3b5d5490d32cc4ca6975cac19422a745607daf8e33abe197"}
Nov 01 06:35:03.039310 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:03.039276    5619 scope.go:122] "RemoveContainer" containerID="a7f486ed0cf861320ee1784afdfcfcedbe18e4345c283cb459fb03c448405a70"
Nov 01 06:35:03.050543 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:03.050524    5619 scope.go:122] "RemoveContainer" containerID="a7f486ed0cf861320ee1784afdfcfcedbe18e4345c283cb459fb03c448405a70"
Nov 01 06:35:03.050844 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:35:03.050817    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7f486ed0cf861320ee1784afdfcfcedbe18e4345c283cb459fb03c448405a70\": container with ID starting with a7f486ed0cf861320ee1784afdfcfcedbe18e4345c283cb459fb03c448405a70 not found: ID does not exist" containerID="a7f486ed0cf861320ee1784afdfcfcedbe18e4345c283cb459fb03c448405a70"
Nov 01 06:35:03.050937 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:03.050853    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7f486ed0cf861320ee1784afdfcfcedbe18e4345c283cb459fb03c448405a70"} err="failed to get container status \"a7f486ed0cf861320ee1784afdfcfcedbe18e4345c283cb459fb03c448405a70\": rpc error: code = NotFound desc = could not find container \"a7f486ed0cf861320ee1784afdfcfcedbe18e4345c283cb459fb03c448405a70\": container with ID starting with a7f486ed0cf861320ee1784afdfcfcedbe18e4345c283cb459fb03c448405a70 not found: ID does not exist"
Nov 01 06:35:03.090316 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:03.090292    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c777bdea-6aeb-48cf-80c7-97f263013624-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:03.090316 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:03.090316    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vjtp\" (UniqueName: \"kubernetes.io/projected/c777bdea-6aeb-48cf-80c7-97f263013624-kube-api-access-4vjtp\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:03.090483 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:03.090347    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c777bdea-6aeb-48cf-80c7-97f263013624-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:03.192181 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:03.192102    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"]
Nov 01 06:35:03.196241 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:03.196213    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fbc98587d-qvn7c"]
Nov 01 06:35:03.737058 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:03.737025    5619 scope.go:122] "RemoveContainer" containerID="afa4a23afcb4cb57ef867d9c8995d2c0bfe82a7c2c12e8b71d1f77a56058877b"
Nov 01 06:35:04.962938 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:04.962904    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c777bdea-6aeb-48cf-80c7-97f263013624" path="/var/lib/kubelet/pods/c777bdea-6aeb-48cf-80c7-97f263013624/volumes"
Nov 01 06:35:06.295372 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:06.295337    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:09.951312 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:09.951264    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"]
Nov 01 06:35:10.686709 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:35:10.686676    5619 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9f8bf4cbae3d: secret "agc-cred-9f8bf4cbae3d" not found
Nov 01 06:35:10.686709 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:35:10.686705    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-76b56d9557-kpktp: secret "agc-cred-9f8bf4cbae3d" not found
Nov 01 06:35:10.686918 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:35:10.686772    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-provider-credentials podName:03c227a0-506e-446e-a0ef-b9650a02b21f nodeName:}" failed. No retries permitted until 2026-11-01 06:35:26.686751484 +0000 UTC m=+2963.897574985 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-provider-credentials") pod "payload-processing-external-model-76b56d9557-kpktp" (UID: "03c227a0-506e-446e-a0ef-b9650a02b21f") : secret "agc-cred-9f8bf4cbae3d" not found
Nov 01 06:35:11.170029 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:11.169995    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:14.952512 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:14.952466    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp" podUID="03c227a0-506e-446e-a0ef-b9650a02b21f" containerName="payload-processing" containerID="cri-o://8c5587441fff843eaac1bdcd2f6d1159f25d977c849ca0beec4ec5a2ee00a4be" gracePeriod=25
Nov 01 06:35:15.120862 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:15.120831    5619 generic.go:356] "Generic (PLEG): container finished" podID="03c227a0-506e-446e-a0ef-b9650a02b21f" containerID="8c5587441fff843eaac1bdcd2f6d1159f25d977c849ca0beec4ec5a2ee00a4be" exitCode=0
Nov 01 06:35:15.121021 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:15.120905    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp" event={"ID":"03c227a0-506e-446e-a0ef-b9650a02b21f","Type":"ContainerDied","Data":"8c5587441fff843eaac1bdcd2f6d1159f25d977c849ca0beec4ec5a2ee00a4be"}
Nov 01 06:35:15.219149 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:15.219089    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"
Nov 01 06:35:15.348553 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:15.347576    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-kube-api-access-q4z87\" (UniqueName: \"kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-kube-api-access-q4z87\") pod \"03c227a0-506e-446e-a0ef-b9650a02b21f\" (UID: \"03c227a0-506e-446e-a0ef-b9650a02b21f\") "
Nov 01 06:35:15.348553 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:15.347647    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03c227a0-506e-446e-a0ef-b9650a02b21f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c227a0-506e-446e-a0ef-b9650a02b21f-plugins-config-volume\") pod \"03c227a0-506e-446e-a0ef-b9650a02b21f\" (UID: \"03c227a0-506e-446e-a0ef-b9650a02b21f\") "
Nov 01 06:35:15.348553 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:15.347763    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-provider-credentials\") pod \"03c227a0-506e-446e-a0ef-b9650a02b21f\" (UID: \"03c227a0-506e-446e-a0ef-b9650a02b21f\") "
Nov 01 06:35:15.348553 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:15.347819    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03c227a0-506e-446e-a0ef-b9650a02b21f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03c227a0-506e-446e-a0ef-b9650a02b21f-routing-overlay-volume\") pod \"03c227a0-506e-446e-a0ef-b9650a02b21f\" (UID: \"03c227a0-506e-446e-a0ef-b9650a02b21f\") "
Nov 01 06:35:15.349111 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:15.349078    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03c227a0-506e-446e-a0ef-b9650a02b21f-routing-overlay-volume" pod "03c227a0-506e-446e-a0ef-b9650a02b21f" (UID: "03c227a0-506e-446e-a0ef-b9650a02b21f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:35:15.349380 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:15.349352    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03c227a0-506e-446e-a0ef-b9650a02b21f-plugins-config-volume" pod "03c227a0-506e-446e-a0ef-b9650a02b21f" (UID: "03c227a0-506e-446e-a0ef-b9650a02b21f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:35:15.351514 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:15.351483    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-provider-credentials" pod "03c227a0-506e-446e-a0ef-b9650a02b21f" (UID: "03c227a0-506e-446e-a0ef-b9650a02b21f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:35:15.351628 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:15.351577    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-kube-api-access-q4z87" pod "03c227a0-506e-446e-a0ef-b9650a02b21f" (UID: "03c227a0-506e-446e-a0ef-b9650a02b21f"). InnerVolumeSpecName "kube-api-access-q4z87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:35:15.449449 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:15.449414    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4z87\" (UniqueName: \"kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-kube-api-access-q4z87\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:15.449449 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:15.449448    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c227a0-506e-446e-a0ef-b9650a02b21f-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:15.449679 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:15.449462    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03c227a0-506e-446e-a0ef-b9650a02b21f-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:15.449679 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:15.449476    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03c227a0-506e-446e-a0ef-b9650a02b21f-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:15.962728 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:15.962691    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:16.130291 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:16.130262    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"
Nov 01 06:35:16.130291 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:16.130282    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76b56d9557-kpktp" event={"ID":"03c227a0-506e-446e-a0ef-b9650a02b21f","Type":"ContainerDied","Data":"4e0d9d49fbd0929256ccbb2f0351c179eb8fa819f2259525ebdc22241e0ff851"}
Nov 01 06:35:16.130571 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:16.130380    5619 scope.go:122] "RemoveContainer" containerID="8c5587441fff843eaac1bdcd2f6d1159f25d977c849ca0beec4ec5a2ee00a4be"
Nov 01 06:35:16.298806 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:16.298725    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"]
Nov 01 06:35:16.302566 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:16.302541    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-76b56d9557-kpktp"]
Nov 01 06:35:16.963499 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:16.963468    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03c227a0-506e-446e-a0ef-b9650a02b21f" path="/var/lib/kubelet/pods/03c227a0-506e-446e-a0ef-b9650a02b21f/volumes"
Nov 01 06:35:20.696164 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:20.696118    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:25.479348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:25.478917    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:25.963931 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:35:25.963812    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:35:27.810304 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:27.810265    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:27.978252 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:27.978217    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:28.550780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:28.550739    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:30.393668 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:30.393623    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:33.217254 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:33.217218    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:34.490534 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:34.490385    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:36.044908 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:36.044868    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:36.258761 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:36.258727    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:36.399281 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:36.399237    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:39.608906 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:39.608871    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"]
Nov 01 06:35:39.756763 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:39.756727    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:39.921234 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:39.921148    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:40.312766 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:40.312685    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:41.276490 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:41.276404    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:44.609727 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:44.609690    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l" podUID="5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1" containerName="payload-processing" containerID="cri-o://0c0763779f13a7eab27a60d9d533ac2bc6a7b0df8dac70100620eefa08e474b2" gracePeriod=25
Nov 01 06:35:44.664592 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:35:44.664567    5619 logging.go:55] [core] [Channel #4238 SubChannel #4239]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.135:50052", ServerName: "10.133.0.135:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.135:50052: connect: connection refused"
Nov 01 06:35:44.874375 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:44.874292    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"
Nov 01 06:35:44.981813 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:44.981784    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"]
Nov 01 06:35:44.982367 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:44.982351    5619 cpu_manager.go:414] "containerMap: removing container" podUID="03c227a0-506e-446e-a0ef-b9650a02b21f" containerName="payload-processing"
Nov 01 06:35:44.982426 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:44.982368    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03c227a0-506e-446e-a0ef-b9650a02b21f" containerName="payload-processing"
Nov 01 06:35:44.982426 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:44.982396    5619 cpu_manager.go:414] "containerMap: removing container" podUID="c777bdea-6aeb-48cf-80c7-97f263013624" containerName="payload-processing"
Nov 01 06:35:44.982426 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:44.982401    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c777bdea-6aeb-48cf-80c7-97f263013624" containerName="payload-processing"
Nov 01 06:35:44.982426 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:44.982415    5619 cpu_manager.go:414] "containerMap: removing container" podUID="5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1" containerName="payload-processing"
Nov 01 06:35:44.982426 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:44.982419    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1" containerName="payload-processing"
Nov 01 06:35:44.982583 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:44.982511    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="c777bdea-6aeb-48cf-80c7-97f263013624" containerName="payload-processing"
Nov 01 06:35:44.982583 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:44.982524    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1" containerName="payload-processing"
Nov 01 06:35:44.982583 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:44.982534    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="03c227a0-506e-446e-a0ef-b9650a02b21f" containerName="payload-processing"
Nov 01 06:35:44.985784 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:44.985767    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:45.043725 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.043693    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-d7d444\"/\"default-dockercfg-rz78w\""
Nov 01 06:35:45.044052 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.044022    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-d7d444\"/\"kube-root-ca.crt\""
Nov 01 06:35:45.044554 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.044527    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-d7d444\"/\"openshift-service-ca.crt\""
Nov 01 06:35:45.045784 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.045761    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-d7d444\"/\"rate-test-model-d7d444-kserve-self-signed-certs\""
Nov 01 06:35:45.047759 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.047737    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"]
Nov 01 06:35:45.059176 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.059149    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-routing-overlay-volume\") pod \"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1\" (UID: \"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1\") "
Nov 01 06:35:45.059302 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.059206    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-provider-credentials\") pod \"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1\" (UID: \"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1\") "
Nov 01 06:35:45.059418 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.059357    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-plugins-config-volume\") pod \"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1\" (UID: \"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1\") "
Nov 01 06:35:45.059600 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.059575    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-routing-overlay-volume" pod "5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1" (UID: "5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:35:45.059726 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.059706    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-plugins-config-volume" pod "5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1" (UID: "5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:35:45.059834 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.059820    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-kube-api-access-ddxts\" (UniqueName: \"kubernetes.io/projected/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-kube-api-access-ddxts\") pod \"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1\" (UID: \"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1\") "
Nov 01 06:35:45.060273 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.060246    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:45.060402 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.060275    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:45.061843 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.061817    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-provider-credentials" pod "5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1" (UID: "5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:35:45.062075 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.062059    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-kube-api-access-ddxts" pod "5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1" (UID: "5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1"). InnerVolumeSpecName "kube-api-access-ddxts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:35:45.090514 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.090482    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:45.161522 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.161433    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spkjd\" (UniqueName: \"kubernetes.io/projected/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-kube-api-access-spkjd\") pod \"rate-test-model-d7d444-kserve-79795bcd69-mbn6j\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:45.161672 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.161545    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-tls-certs\") pod \"rate-test-model-d7d444-kserve-79795bcd69-mbn6j\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:45.161672 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.161613    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-home\") pod \"rate-test-model-d7d444-kserve-79795bcd69-mbn6j\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:45.161761 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.161715    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-model-cache\") pod \"rate-test-model-d7d444-kserve-79795bcd69-mbn6j\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:45.161818 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.161803    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-tmp-dir\") pod \"rate-test-model-d7d444-kserve-79795bcd69-mbn6j\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:45.161868 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.161859    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-dshm\") pod \"rate-test-model-d7d444-kserve-79795bcd69-mbn6j\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:45.161992 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.161979    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddxts\" (UniqueName: \"kubernetes.io/projected/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-kube-api-access-ddxts\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:45.162039 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.161996    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:35:45.262692 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.262655    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-home\") pod \"rate-test-model-d7d444-kserve-79795bcd69-mbn6j\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:45.262885 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.262702    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-model-cache\") pod \"rate-test-model-d7d444-kserve-79795bcd69-mbn6j\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:45.262885 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.262731    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-tmp-dir\") pod \"rate-test-model-d7d444-kserve-79795bcd69-mbn6j\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:45.262885 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.262802    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-dshm\") pod \"rate-test-model-d7d444-kserve-79795bcd69-mbn6j\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:45.263065 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.262883    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spkjd\" (UniqueName: \"kubernetes.io/projected/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-kube-api-access-spkjd\") pod \"rate-test-model-d7d444-kserve-79795bcd69-mbn6j\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:45.263065 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.262965    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-tls-certs\") pod \"rate-test-model-d7d444-kserve-79795bcd69-mbn6j\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:45.263185 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.263069    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-model-cache\") pod \"rate-test-model-d7d444-kserve-79795bcd69-mbn6j\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:45.263185 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.263113    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-home\") pod \"rate-test-model-d7d444-kserve-79795bcd69-mbn6j\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:45.263185 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.263175    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-tmp-dir\") pod \"rate-test-model-d7d444-kserve-79795bcd69-mbn6j\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:45.265119 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.265093    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-dshm\") pod \"rate-test-model-d7d444-kserve-79795bcd69-mbn6j\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:45.265374 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.265355    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-tls-certs\") pod \"rate-test-model-d7d444-kserve-79795bcd69-mbn6j\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:45.323210 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.323178    5619 generic.go:356] "Generic (PLEG): container finished" podID="5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1" containerID="0c0763779f13a7eab27a60d9d533ac2bc6a7b0df8dac70100620eefa08e474b2" exitCode=0
Nov 01 06:35:45.323367 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.323212    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l" event={"ID":"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1","Type":"ContainerDied","Data":"0c0763779f13a7eab27a60d9d533ac2bc6a7b0df8dac70100620eefa08e474b2"}
Nov 01 06:35:45.323367 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.323225    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"
Nov 01 06:35:45.323367 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.323269    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l" event={"ID":"5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1","Type":"ContainerDied","Data":"39ac297d84531419588e5e7dcff779cadf6bd5774f9b009c466462f82c2c6857"}
Nov 01 06:35:45.323367 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.323296    5619 scope.go:122] "RemoveContainer" containerID="0c0763779f13a7eab27a60d9d533ac2bc6a7b0df8dac70100620eefa08e474b2"
Nov 01 06:35:45.324866 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.324843    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spkjd\" (UniqueName: \"kubernetes.io/projected/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-kube-api-access-spkjd\") pod \"rate-test-model-d7d444-kserve-79795bcd69-mbn6j\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") " pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:45.333141 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.333126    5619 scope.go:122] "RemoveContainer" containerID="0c0763779f13a7eab27a60d9d533ac2bc6a7b0df8dac70100620eefa08e474b2"
Nov 01 06:35:45.333435 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:35:45.333408    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c0763779f13a7eab27a60d9d533ac2bc6a7b0df8dac70100620eefa08e474b2\": container with ID starting with 0c0763779f13a7eab27a60d9d533ac2bc6a7b0df8dac70100620eefa08e474b2 not found: ID does not exist" containerID="0c0763779f13a7eab27a60d9d533ac2bc6a7b0df8dac70100620eefa08e474b2"
Nov 01 06:35:45.333540 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.333446    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0763779f13a7eab27a60d9d533ac2bc6a7b0df8dac70100620eefa08e474b2"} err="failed to get container status \"0c0763779f13a7eab27a60d9d533ac2bc6a7b0df8dac70100620eefa08e474b2\": rpc error: code = NotFound desc = could not find container \"0c0763779f13a7eab27a60d9d533ac2bc6a7b0df8dac70100620eefa08e474b2\": container with ID starting with 0c0763779f13a7eab27a60d9d533ac2bc6a7b0df8dac70100620eefa08e474b2 not found: ID does not exist"
Nov 01 06:35:45.492909 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.492818    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"]
Nov 01 06:35:45.496313 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.496268    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l"]
Nov 01 06:35:45.598035 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.597995    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:45.675945 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.674629    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-cfe368/payload-processing-external-model-e2e-extmod-a-cfe368-846bxgf9l" podUID="5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.135:50052\" within 1s: context deadline exceeded"
Nov 01 06:35:45.675945 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:35:45.674876    5619 logging.go:55] [core] [Channel #4238 SubChannel #4239]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.135:50052", ServerName: "10.133.0.135:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.135:50052: operation was canceled"
Nov 01 06:35:45.983094 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:45.983065    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"]
Nov 01 06:35:45.984084 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:35:45.984053    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81cdf4f9_7e3c_47c0_b669_e06ec353fdf6.slice/crio-7ba62fbc669343de11d4aacd3e5b740aecdcd3ddfe7a651ebfd521fae9727bc4 WatchSource:0}: Error finding container 7ba62fbc669343de11d4aacd3e5b740aecdcd3ddfe7a651ebfd521fae9727bc4: Status 404 returned error can't find the container with id 7ba62fbc669343de11d4aacd3e5b740aecdcd3ddfe7a651ebfd521fae9727bc4
Nov 01 06:35:46.169111 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:46.169066    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:46.335075 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:46.334986    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j" event={"ID":"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6","Type":"ContainerStarted","Data":"64b602c1bd3dfc3eafb5a2b92d2a57b2039ffe1e5242d07f85e46f6065047b68"}
Nov 01 06:35:46.335075 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:46.335030    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j" event={"ID":"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6","Type":"ContainerStarted","Data":"7ba62fbc669343de11d4aacd3e5b740aecdcd3ddfe7a651ebfd521fae9727bc4"}
Nov 01 06:35:46.454837 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:46.454783    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j" podStartSLOduration=2.454769106 podStartE2EDuration="2.454769106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:35:46.452771646 +0000 UTC m=+2983.663595158" watchObservedRunningTime="2026-11-01 06:35:46.454769106 +0000 UTC m=+2983.665592636"
Nov 01 06:35:46.598603 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:46.598570    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:46.968839 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:46.968751    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1" path="/var/lib/kubelet/pods/5a0a06c3-710b-4eba-95d1-e6fd2b70f1c1/volumes"
Nov 01 06:35:50.654515 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:50.654465    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:50.955049 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:50.954956    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:54.233293 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:54.233248    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:54.430076 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:54.430037    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:55.598604 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:55.598572    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:55.611806 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:55.611777    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:55.883953 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:35:55.883916    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:35:55.958475 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:55.958431    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:56.422064 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:56.422032    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:35:56.535729 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:56.535695    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:35:56.734894 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:35:56.734801    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:00.966958 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:00.966913    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:01.893442 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:01.893406    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:02.090747 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:02.090708    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:05.780665 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:05.780627    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:09.007165 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:09.007129    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:09.461719 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:09.461678    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:10.557415 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:10.557382    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:14.513035 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:14.513000    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:15.262235 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:15.262198    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:20.049929 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:20.049892    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:24.716732 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:24.716690    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:24.990706 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:24.990625    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:25.138657 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:25.138617    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:25.881639 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:36:25.881599    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:36:28.908896 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:28.908848    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:29.265483 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:29.265393    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"]
Nov 01 06:36:29.270873 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:29.270847    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"
Nov 01 06:36:29.324295 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:29.324265    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"agc-cred-a97b9b6eb74d\""
Nov 01 06:36:29.377920 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:29.377884    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"]
Nov 01 06:36:29.417766 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:29.417722    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9\" (UID: \"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"
Nov 01 06:36:29.417945 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:29.417772    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9\" (UID: \"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"
Nov 01 06:36:29.417945 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:29.417923    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz4td\" (UniqueName: \"kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-kube-api-access-xz4td\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9\" (UID: \"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"
Nov 01 06:36:29.418071 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:29.418001    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9\" (UID: \"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"
Nov 01 06:36:29.519502 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:29.519401    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9\" (UID: \"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"
Nov 01 06:36:29.519502 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:29.519438    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9\" (UID: \"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"
Nov 01 06:36:29.519760 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:29.519506    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz4td\" (UniqueName: \"kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-kube-api-access-xz4td\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9\" (UID: \"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"
Nov 01 06:36:29.519760 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:29.519582    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9\" (UID: \"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"
Nov 01 06:36:29.520212 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:29.520186    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9\" (UID: \"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"
Nov 01 06:36:29.520313 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:29.520212    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9\" (UID: \"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"
Nov 01 06:36:29.522311 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:29.522287    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9\" (UID: \"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"
Nov 01 06:36:29.548745 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:29.548713    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:29.581270 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:29.581246    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz4td\" (UniqueName: \"kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-kube-api-access-xz4td\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9\" (UID: \"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"
Nov 01 06:36:29.584686 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:29.584666    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"
Nov 01 06:36:29.979651 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:29.979618    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"]
Nov 01 06:36:29.980942 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:36:29.980916    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7afa361c_dc3c_4281_b43b_c7d4a7dfcac1.slice/crio-d9b3c258f6d9263f0767fc76b959b6fcd83f7221b48dafda3aa07f4f4170103a WatchSource:0}: Error finding container d9b3c258f6d9263f0767fc76b959b6fcd83f7221b48dafda3aa07f4f4170103a: Status 404 returned error can't find the container with id d9b3c258f6d9263f0767fc76b959b6fcd83f7221b48dafda3aa07f4f4170103a
Nov 01 06:36:30.640340 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:30.640290    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9" event={"ID":"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1","Type":"ContainerStarted","Data":"3c5fdc6903820c5a2db272f4dc1abede3138f5c423db3dc85481e0a5afa74a16"}
Nov 01 06:36:30.640539 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:30.640370    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9" event={"ID":"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1","Type":"ContainerStarted","Data":"d9b3c258f6d9263f0767fc76b959b6fcd83f7221b48dafda3aa07f4f4170103a"}
Nov 01 06:36:30.640539 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:30.640399    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"
Nov 01 06:36:30.758947 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:30.758896    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9" podStartSLOduration=1.758882094 podStartE2EDuration="1.758882094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:36:30.757741169 +0000 UTC m=+3027.968564680" watchObservedRunningTime="2026-11-01 06:36:30.758882094 +0000 UTC m=+3027.969705602"
Nov 01 06:36:33.569442 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:33.569404    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:34.257593 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:34.257560    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:38.693461 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:38.693428    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"]
Nov 01 06:36:38.697615 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:38.697589    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"
Nov 01 06:36:38.724391 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:38.724363    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvf6c\" (UniqueName: \"kubernetes.io/projected/304878ad-a41a-4349-8ec7-3c7d38f4efbc-kube-api-access-rvf6c\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg\" (UID: \"304878ad-a41a-4349-8ec7-3c7d38f4efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"
Nov 01 06:36:38.724489 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:38.724396    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/304878ad-a41a-4349-8ec7-3c7d38f4efbc-openshift-service-ca\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg\" (UID: \"304878ad-a41a-4349-8ec7-3c7d38f4efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"
Nov 01 06:36:38.724577 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:38.724553    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/304878ad-a41a-4349-8ec7-3c7d38f4efbc-maas-api-tls\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg\" (UID: \"304878ad-a41a-4349-8ec7-3c7d38f4efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"
Nov 01 06:36:38.724617 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:38.724610    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/304878ad-a41a-4349-8ec7-3c7d38f4efbc-metrics-certs\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg\" (UID: \"304878ad-a41a-4349-8ec7-3c7d38f4efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"
Nov 01 06:36:38.750640 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:38.750615    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c0662f25-serving-cert\""
Nov 01 06:36:38.798952 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:38.798924    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"]
Nov 01 06:36:38.826019 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:38.825990    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/304878ad-a41a-4349-8ec7-3c7d38f4efbc-maas-api-tls\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg\" (UID: \"304878ad-a41a-4349-8ec7-3c7d38f4efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"
Nov 01 06:36:38.826198 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:38.826054    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/304878ad-a41a-4349-8ec7-3c7d38f4efbc-metrics-certs\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg\" (UID: \"304878ad-a41a-4349-8ec7-3c7d38f4efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"
Nov 01 06:36:38.826198 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:38.826142    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvf6c\" (UniqueName: \"kubernetes.io/projected/304878ad-a41a-4349-8ec7-3c7d38f4efbc-kube-api-access-rvf6c\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg\" (UID: \"304878ad-a41a-4349-8ec7-3c7d38f4efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"
Nov 01 06:36:38.826198 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:38.826161    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/304878ad-a41a-4349-8ec7-3c7d38f4efbc-openshift-service-ca\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg\" (UID: \"304878ad-a41a-4349-8ec7-3c7d38f4efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"
Nov 01 06:36:38.826907 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:38.826875    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/304878ad-a41a-4349-8ec7-3c7d38f4efbc-openshift-service-ca\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg\" (UID: \"304878ad-a41a-4349-8ec7-3c7d38f4efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"
Nov 01 06:36:38.828526 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:36:38.828454    5619 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-serving-cert: secret "maas-api-e2e-mt-c0662f25-serving-cert" not found
Nov 01 06:36:38.828863 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:36:38.828841    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/304878ad-a41a-4349-8ec7-3c7d38f4efbc-maas-api-tls podName:304878ad-a41a-4349-8ec7-3c7d38f4efbc nodeName:}" failed. No retries permitted until 2026-11-01 06:36:39.328811979 +0000 UTC m=+3036.539635482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/304878ad-a41a-4349-8ec7-3c7d38f4efbc-maas-api-tls") pod "maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg" (UID: "304878ad-a41a-4349-8ec7-3c7d38f4efbc") : secret "maas-api-e2e-mt-c0662f25-serving-cert" not found
Nov 01 06:36:38.830419 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:38.830397    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/304878ad-a41a-4349-8ec7-3c7d38f4efbc-metrics-certs\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg\" (UID: \"304878ad-a41a-4349-8ec7-3c7d38f4efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"
Nov 01 06:36:38.888790 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:38.888753    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvf6c\" (UniqueName: \"kubernetes.io/projected/304878ad-a41a-4349-8ec7-3c7d38f4efbc-kube-api-access-rvf6c\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg\" (UID: \"304878ad-a41a-4349-8ec7-3c7d38f4efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"
Nov 01 06:36:38.892309 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:38.892276    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:39.091837 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:39.091803    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:39.239680 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:39.239641    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:39.332671 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:39.332636    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/304878ad-a41a-4349-8ec7-3c7d38f4efbc-maas-api-tls\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg\" (UID: \"304878ad-a41a-4349-8ec7-3c7d38f4efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"
Nov 01 06:36:39.334874 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:39.334842    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/304878ad-a41a-4349-8ec7-3c7d38f4efbc-maas-api-tls\") pod \"maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg\" (UID: \"304878ad-a41a-4349-8ec7-3c7d38f4efbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"
Nov 01 06:36:39.611525 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:39.611494    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"
Nov 01 06:36:40.004142 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:40.004115    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"]
Nov 01 06:36:40.005629 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:36:40.005594    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod304878ad_a41a_4349_8ec7_3c7d38f4efbc.slice/crio-2b9a8deaf5fcb43a777d31d9dd5efbe71194a2b8128421fdfd629a1c3523adac WatchSource:0}: Error finding container 2b9a8deaf5fcb43a777d31d9dd5efbe71194a2b8128421fdfd629a1c3523adac: Status 404 returned error can't find the container with id 2b9a8deaf5fcb43a777d31d9dd5efbe71194a2b8128421fdfd629a1c3523adac
Nov 01 06:36:40.006977 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:40.006960    5619 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 06:36:40.711369 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:40.711334    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg" event={"ID":"304878ad-a41a-4349-8ec7-3c7d38f4efbc","Type":"ContainerStarted","Data":"2b9a8deaf5fcb43a777d31d9dd5efbe71194a2b8128421fdfd629a1c3523adac"}
Nov 01 06:36:41.649146 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:41.649119    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"
Nov 01 06:36:41.719469 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:41.719434    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg" event={"ID":"304878ad-a41a-4349-8ec7-3c7d38f4efbc","Type":"ContainerStarted","Data":"755c5e24ef849bd332dd24b08430b76f06c324409b16ce24eb8b3b82b3400fc4"}
Nov 01 06:36:41.719625 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:41.719537    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"
Nov 01 06:36:41.821390 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:41.821358    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"]
Nov 01 06:36:41.891117 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:41.891061    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg" podStartSLOduration=2.896692635 podStartE2EDuration="3.8910473s" totalImagesPullingTime="994.354665ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:36:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:36:41.887855253 +0000 UTC m=+3039.098678798" watchObservedRunningTime="2026-11-01 06:36:41.8910473 +0000 UTC m=+3039.101870808"
Nov 01 06:36:43.858526 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:43.858470    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:46.821751 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:46.821713    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb" podUID="75319d6b-af9b-4c2f-a53d-71b2a64f3325" containerName="payload-processing" containerID="cri-o://72e3023d326aaa7932abfa30307b5ebd93c279f94f0a5212dfa8a73d0e7a69a5" gracePeriod=25
Nov 01 06:36:47.091544 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.091520    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"
Nov 01 06:36:47.229693 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.229662    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75319d6b-af9b-4c2f-a53d-71b2a64f3325-provider-credentials\" (UniqueName: \"kubernetes.io/projected/75319d6b-af9b-4c2f-a53d-71b2a64f3325-provider-credentials\") pod \"75319d6b-af9b-4c2f-a53d-71b2a64f3325\" (UID: \"75319d6b-af9b-4c2f-a53d-71b2a64f3325\") "
Nov 01 06:36:47.229891 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.229714    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75319d6b-af9b-4c2f-a53d-71b2a64f3325-kube-api-access-csxjx\" (UniqueName: \"kubernetes.io/projected/75319d6b-af9b-4c2f-a53d-71b2a64f3325-kube-api-access-csxjx\") pod \"75319d6b-af9b-4c2f-a53d-71b2a64f3325\" (UID: \"75319d6b-af9b-4c2f-a53d-71b2a64f3325\") "
Nov 01 06:36:47.229891 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.229859    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75319d6b-af9b-4c2f-a53d-71b2a64f3325-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75319d6b-af9b-4c2f-a53d-71b2a64f3325-routing-overlay-volume\") pod \"75319d6b-af9b-4c2f-a53d-71b2a64f3325\" (UID: \"75319d6b-af9b-4c2f-a53d-71b2a64f3325\") "
Nov 01 06:36:47.230031 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.229996    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75319d6b-af9b-4c2f-a53d-71b2a64f3325-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75319d6b-af9b-4c2f-a53d-71b2a64f3325-plugins-config-volume\") pod \"75319d6b-af9b-4c2f-a53d-71b2a64f3325\" (UID: \"75319d6b-af9b-4c2f-a53d-71b2a64f3325\") "
Nov 01 06:36:47.230654 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.230199    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75319d6b-af9b-4c2f-a53d-71b2a64f3325-routing-overlay-volume" pod "75319d6b-af9b-4c2f-a53d-71b2a64f3325" (UID: "75319d6b-af9b-4c2f-a53d-71b2a64f3325"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:36:47.230654 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.230536    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75319d6b-af9b-4c2f-a53d-71b2a64f3325-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:36:47.230654 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.230612    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75319d6b-af9b-4c2f-a53d-71b2a64f3325-plugins-config-volume" pod "75319d6b-af9b-4c2f-a53d-71b2a64f3325" (UID: "75319d6b-af9b-4c2f-a53d-71b2a64f3325"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:36:47.234576 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.234550    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75319d6b-af9b-4c2f-a53d-71b2a64f3325-kube-api-access-csxjx" pod "75319d6b-af9b-4c2f-a53d-71b2a64f3325" (UID: "75319d6b-af9b-4c2f-a53d-71b2a64f3325"). InnerVolumeSpecName "kube-api-access-csxjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:36:47.240515 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.240486    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75319d6b-af9b-4c2f-a53d-71b2a64f3325-provider-credentials" pod "75319d6b-af9b-4c2f-a53d-71b2a64f3325" (UID: "75319d6b-af9b-4c2f-a53d-71b2a64f3325"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:36:47.332157 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.332124    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75319d6b-af9b-4c2f-a53d-71b2a64f3325-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:36:47.332157 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.332154    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75319d6b-af9b-4c2f-a53d-71b2a64f3325-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:36:47.332157 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.332165    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csxjx\" (UniqueName: \"kubernetes.io/projected/75319d6b-af9b-4c2f-a53d-71b2a64f3325-kube-api-access-csxjx\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:36:47.728118 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.728078    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg" podUID="304878ad-a41a-4349-8ec7-3c7d38f4efbc" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.139:8443/health\": dial tcp 10.133.0.139:8443: connect: connection refused"
Nov 01 06:36:47.760262 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.760229    5619 generic.go:356] "Generic (PLEG): container finished" podID="75319d6b-af9b-4c2f-a53d-71b2a64f3325" containerID="72e3023d326aaa7932abfa30307b5ebd93c279f94f0a5212dfa8a73d0e7a69a5" exitCode=0
Nov 01 06:36:47.760262 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.760271    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"
Nov 01 06:36:47.760500 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.760303    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb" event={"ID":"75319d6b-af9b-4c2f-a53d-71b2a64f3325","Type":"ContainerDied","Data":"72e3023d326aaa7932abfa30307b5ebd93c279f94f0a5212dfa8a73d0e7a69a5"}
Nov 01 06:36:47.760500 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.760380    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb" event={"ID":"75319d6b-af9b-4c2f-a53d-71b2a64f3325","Type":"ContainerDied","Data":"dae8ada0e98cc0a9d0cc141404a1b70aa82f135395df46444a86545a38ae95c7"}
Nov 01 06:36:47.760500 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.760402    5619 scope.go:122] "RemoveContainer" containerID="72e3023d326aaa7932abfa30307b5ebd93c279f94f0a5212dfa8a73d0e7a69a5"
Nov 01 06:36:47.770232 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.770209    5619 scope.go:122] "RemoveContainer" containerID="72e3023d326aaa7932abfa30307b5ebd93c279f94f0a5212dfa8a73d0e7a69a5"
Nov 01 06:36:47.770519 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:36:47.770492    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72e3023d326aaa7932abfa30307b5ebd93c279f94f0a5212dfa8a73d0e7a69a5\": container with ID starting with 72e3023d326aaa7932abfa30307b5ebd93c279f94f0a5212dfa8a73d0e7a69a5 not found: ID does not exist" containerID="72e3023d326aaa7932abfa30307b5ebd93c279f94f0a5212dfa8a73d0e7a69a5"
Nov 01 06:36:47.770580 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.770530    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72e3023d326aaa7932abfa30307b5ebd93c279f94f0a5212dfa8a73d0e7a69a5"} err="failed to get container status \"72e3023d326aaa7932abfa30307b5ebd93c279f94f0a5212dfa8a73d0e7a69a5\": rpc error: code = NotFound desc = could not find container \"72e3023d326aaa7932abfa30307b5ebd93c279f94f0a5212dfa8a73d0e7a69a5\": container with ID starting with 72e3023d326aaa7932abfa30307b5ebd93c279f94f0a5212dfa8a73d0e7a69a5 not found: ID does not exist"
Nov 01 06:36:47.928465 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.928427    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"]
Nov 01 06:36:47.978886 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:47.978803    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598sxqfb"]
Nov 01 06:36:48.668512 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:48.668478    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:48.962380 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:48.962297    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75319d6b-af9b-4c2f-a53d-71b2a64f3325" path="/var/lib/kubelet/pods/75319d6b-af9b-4c2f-a53d-71b2a64f3325/volumes"
Nov 01 06:36:51.564739 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:51.564643    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:51.787826 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:51.787794    5619 generic.go:356] "Generic (PLEG): container finished" podID="304878ad-a41a-4349-8ec7-3c7d38f4efbc" containerID="755c5e24ef849bd332dd24b08430b76f06c324409b16ce24eb8b3b82b3400fc4" exitCode=1
Nov 01 06:36:51.787994 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:51.787868    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg" event={"ID":"304878ad-a41a-4349-8ec7-3c7d38f4efbc","Type":"ContainerDied","Data":"755c5e24ef849bd332dd24b08430b76f06c324409b16ce24eb8b3b82b3400fc4"}
Nov 01 06:36:51.788281 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:51.788267    5619 scope.go:122] "RemoveContainer" containerID="755c5e24ef849bd332dd24b08430b76f06c324409b16ce24eb8b3b82b3400fc4"
Nov 01 06:36:52.329536 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:52.329500    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67"]
Nov 01 06:36:52.441365 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:52.441336    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw"]
Nov 01 06:36:52.718540 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:52.718507    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:52.798843 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:52.798804    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg" event={"ID":"304878ad-a41a-4349-8ec7-3c7d38f4efbc","Type":"ContainerStarted","Data":"e4acad68fd3fa2ac56fecab8153758fd274f0d25ec67d1ee3d5ed8158dbad2fa"}
Nov 01 06:36:52.799067 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:52.799049    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"
Nov 01 06:36:52.910833 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:52.910742    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:53.660780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:53.660744    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:55.879539 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:36:55.879501    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:36:57.095529 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.095493    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:57.330361 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.330305    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67" podUID="c20bff8c-c80c-4776-87d1-7dc271ae1dc7" containerName="payload-processing" containerID="cri-o://2e2b7b5926ee71cb7ea04333d0a192ff6609d502ab5534d3cc78780ca536c72d" gracePeriod=25
Nov 01 06:36:57.441908 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.441876    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw" podUID="c86eab09-9ed0-4dc3-9b5e-65b59abef247" containerName="payload-pre-processing" containerID="cri-o://f03b8ba0561ce12d9c2cd821f7a40dded972a9df9d225847522bdf34bef34d71" gracePeriod=25
Nov 01 06:36:57.611678 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.611614    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67"
Nov 01 06:36:57.655876 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.655837    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c20bff8c-c80c-4776-87d1-7dc271ae1dc7-kube-api-access-74flp\" (UniqueName: \"kubernetes.io/projected/c20bff8c-c80c-4776-87d1-7dc271ae1dc7-kube-api-access-74flp\") pod \"c20bff8c-c80c-4776-87d1-7dc271ae1dc7\" (UID: \"c20bff8c-c80c-4776-87d1-7dc271ae1dc7\") "
Nov 01 06:36:57.656018 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.655924    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c20bff8c-c80c-4776-87d1-7dc271ae1dc7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c20bff8c-c80c-4776-87d1-7dc271ae1dc7-plugins-config-volume\") pod \"c20bff8c-c80c-4776-87d1-7dc271ae1dc7\" (UID: \"c20bff8c-c80c-4776-87d1-7dc271ae1dc7\") "
Nov 01 06:36:57.657146 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.657115    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c20bff8c-c80c-4776-87d1-7dc271ae1dc7-plugins-config-volume" pod "c20bff8c-c80c-4776-87d1-7dc271ae1dc7" (UID: "c20bff8c-c80c-4776-87d1-7dc271ae1dc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:36:57.659075 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.659047    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c20bff8c-c80c-4776-87d1-7dc271ae1dc7-kube-api-access-74flp" pod "c20bff8c-c80c-4776-87d1-7dc271ae1dc7" (UID: "c20bff8c-c80c-4776-87d1-7dc271ae1dc7"). InnerVolumeSpecName "kube-api-access-74flp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:36:57.714401 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.714376    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw"
Nov 01 06:36:57.757295 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.757261    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c86eab09-9ed0-4dc3-9b5e-65b59abef247-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86eab09-9ed0-4dc3-9b5e-65b59abef247-plugins-config-volume\") pod \"c86eab09-9ed0-4dc3-9b5e-65b59abef247\" (UID: \"c86eab09-9ed0-4dc3-9b5e-65b59abef247\") "
Nov 01 06:36:57.757501 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.757480    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c86eab09-9ed0-4dc3-9b5e-65b59abef247-kube-api-access-q4nrv\" (UniqueName: \"kubernetes.io/projected/c86eab09-9ed0-4dc3-9b5e-65b59abef247-kube-api-access-q4nrv\") pod \"c86eab09-9ed0-4dc3-9b5e-65b59abef247\" (UID: \"c86eab09-9ed0-4dc3-9b5e-65b59abef247\") "
Nov 01 06:36:57.757737 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.757707    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c86eab09-9ed0-4dc3-9b5e-65b59abef247-plugins-config-volume" pod "c86eab09-9ed0-4dc3-9b5e-65b59abef247" (UID: "c86eab09-9ed0-4dc3-9b5e-65b59abef247"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:36:57.757957 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.757938    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86eab09-9ed0-4dc3-9b5e-65b59abef247-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:36:57.758023 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.757963    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74flp\" (UniqueName: \"kubernetes.io/projected/c20bff8c-c80c-4776-87d1-7dc271ae1dc7-kube-api-access-74flp\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:36:57.758023 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.757980    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c20bff8c-c80c-4776-87d1-7dc271ae1dc7-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:36:57.760152 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.760124    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86eab09-9ed0-4dc3-9b5e-65b59abef247-kube-api-access-q4nrv" pod "c86eab09-9ed0-4dc3-9b5e-65b59abef247" (UID: "c86eab09-9ed0-4dc3-9b5e-65b59abef247"). InnerVolumeSpecName "kube-api-access-q4nrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:36:57.834631 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.834564    5619 generic.go:356] "Generic (PLEG): container finished" podID="c20bff8c-c80c-4776-87d1-7dc271ae1dc7" containerID="2e2b7b5926ee71cb7ea04333d0a192ff6609d502ab5534d3cc78780ca536c72d" exitCode=0
Nov 01 06:36:57.834631 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.834610    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67"
Nov 01 06:36:57.834851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.834654    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67" event={"ID":"c20bff8c-c80c-4776-87d1-7dc271ae1dc7","Type":"ContainerDied","Data":"2e2b7b5926ee71cb7ea04333d0a192ff6609d502ab5534d3cc78780ca536c72d"}
Nov 01 06:36:57.834851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.834721    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67" event={"ID":"c20bff8c-c80c-4776-87d1-7dc271ae1dc7","Type":"ContainerDied","Data":"80aa4ea35b7e73648c3fc40d4bb18948ebba766e4414221f08a51195e8346c4a"}
Nov 01 06:36:57.834851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.834758    5619 scope.go:122] "RemoveContainer" containerID="2e2b7b5926ee71cb7ea04333d0a192ff6609d502ab5534d3cc78780ca536c72d"
Nov 01 06:36:57.836151 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.836124    5619 generic.go:356] "Generic (PLEG): container finished" podID="c86eab09-9ed0-4dc3-9b5e-65b59abef247" containerID="f03b8ba0561ce12d9c2cd821f7a40dded972a9df9d225847522bdf34bef34d71" exitCode=0
Nov 01 06:36:57.836281 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.836167    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw"
Nov 01 06:36:57.836281 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.836201    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw" event={"ID":"c86eab09-9ed0-4dc3-9b5e-65b59abef247","Type":"ContainerDied","Data":"f03b8ba0561ce12d9c2cd821f7a40dded972a9df9d225847522bdf34bef34d71"}
Nov 01 06:36:57.836281 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.836237    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw" event={"ID":"c86eab09-9ed0-4dc3-9b5e-65b59abef247","Type":"ContainerDied","Data":"e2f49b54ca92faa0b51274f0cc7476c08030baf4150f5bceb86557347c0437c8"}
Nov 01 06:36:57.845878 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.845858    5619 scope.go:122] "RemoveContainer" containerID="2e2b7b5926ee71cb7ea04333d0a192ff6609d502ab5534d3cc78780ca536c72d"
Nov 01 06:36:57.847094 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:36:57.847063    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2b7b5926ee71cb7ea04333d0a192ff6609d502ab5534d3cc78780ca536c72d\": container with ID starting with 2e2b7b5926ee71cb7ea04333d0a192ff6609d502ab5534d3cc78780ca536c72d not found: ID does not exist" containerID="2e2b7b5926ee71cb7ea04333d0a192ff6609d502ab5534d3cc78780ca536c72d"
Nov 01 06:36:57.847198 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.847104    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2b7b5926ee71cb7ea04333d0a192ff6609d502ab5534d3cc78780ca536c72d"} err="failed to get container status \"2e2b7b5926ee71cb7ea04333d0a192ff6609d502ab5534d3cc78780ca536c72d\": rpc error: code = NotFound desc = could not find container \"2e2b7b5926ee71cb7ea04333d0a192ff6609d502ab5534d3cc78780ca536c72d\": container with ID starting with 2e2b7b5926ee71cb7ea04333d0a192ff6609d502ab5534d3cc78780ca536c72d not found: ID does not exist"
Nov 01 06:36:57.847198 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.847126    5619 scope.go:122] "RemoveContainer" containerID="f03b8ba0561ce12d9c2cd821f7a40dded972a9df9d225847522bdf34bef34d71"
Nov 01 06:36:57.859218 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.859188    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4nrv\" (UniqueName: \"kubernetes.io/projected/c86eab09-9ed0-4dc3-9b5e-65b59abef247-kube-api-access-q4nrv\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:36:57.864999 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.864940    5619 scope.go:122] "RemoveContainer" containerID="f03b8ba0561ce12d9c2cd821f7a40dded972a9df9d225847522bdf34bef34d71"
Nov 01 06:36:57.865290 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:36:57.865256    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f03b8ba0561ce12d9c2cd821f7a40dded972a9df9d225847522bdf34bef34d71\": container with ID starting with f03b8ba0561ce12d9c2cd821f7a40dded972a9df9d225847522bdf34bef34d71 not found: ID does not exist" containerID="f03b8ba0561ce12d9c2cd821f7a40dded972a9df9d225847522bdf34bef34d71"
Nov 01 06:36:57.865400 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.865291    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f03b8ba0561ce12d9c2cd821f7a40dded972a9df9d225847522bdf34bef34d71"} err="failed to get container status \"f03b8ba0561ce12d9c2cd821f7a40dded972a9df9d225847522bdf34bef34d71\": rpc error: code = NotFound desc = could not find container \"f03b8ba0561ce12d9c2cd821f7a40dded972a9df9d225847522bdf34bef34d71\": container with ID starting with f03b8ba0561ce12d9c2cd821f7a40dded972a9df9d225847522bdf34bef34d71 not found: ID does not exist"
Nov 01 06:36:57.963796 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.963757    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67"]
Nov 01 06:36:57.967656 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:57.967624    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3680c5-77c5d754b7-ptx67"]
Nov 01 06:36:58.131444 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:58.131366    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw"]
Nov 01 06:36:58.135889 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:58.135862    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3680c5-6b579858f5-vbrkw"]
Nov 01 06:36:58.578660 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:58.578578    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:36:58.811171 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:58.811139    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg" podUID="304878ad-a41a-4349-8ec7-3c7d38f4efbc" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.139:8443/health\": dial tcp 10.133.0.139:8443: connect: connection refused"
Nov 01 06:36:58.963049 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:58.963020    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c20bff8c-c80c-4776-87d1-7dc271ae1dc7" path="/var/lib/kubelet/pods/c20bff8c-c80c-4776-87d1-7dc271ae1dc7/volumes"
Nov 01 06:36:58.963478 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:58.963463    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c86eab09-9ed0-4dc3-9b5e-65b59abef247" path="/var/lib/kubelet/pods/c86eab09-9ed0-4dc3-9b5e-65b59abef247/volumes"
Nov 01 06:36:59.977697 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:36:59.977651    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:37:02.872280 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:02.872250    5619 generic.go:356] "Generic (PLEG): container finished" podID="304878ad-a41a-4349-8ec7-3c7d38f4efbc" containerID="e4acad68fd3fa2ac56fecab8153758fd274f0d25ec67d1ee3d5ed8158dbad2fa" exitCode=1
Nov 01 06:37:02.872754 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:02.872291    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg" event={"ID":"304878ad-a41a-4349-8ec7-3c7d38f4efbc","Type":"ContainerDied","Data":"e4acad68fd3fa2ac56fecab8153758fd274f0d25ec67d1ee3d5ed8158dbad2fa"}
Nov 01 06:37:02.872754 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:02.872351    5619 scope.go:122] "RemoveContainer" containerID="755c5e24ef849bd332dd24b08430b76f06c324409b16ce24eb8b3b82b3400fc4"
Nov 01 06:37:02.872891 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:02.872785    5619 scope.go:122] "RemoveContainer" containerID="e4acad68fd3fa2ac56fecab8153758fd274f0d25ec67d1ee3d5ed8158dbad2fa"
Nov 01 06:37:02.873119 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:37:02.873090    5619 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg_odh-ai-gateway-infra(304878ad-a41a-4349-8ec7-3c7d38f4efbc)\"" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg" podUID="304878ad-a41a-4349-8ec7-3c7d38f4efbc"
Nov 01 06:37:03.424483 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:03.424449    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:37:08.081083 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:08.081047    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:37:09.612377 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:09.612318    5619 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"
Nov 01 06:37:09.612770 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:09.612755    5619 scope.go:122] "RemoveContainer" containerID="e4acad68fd3fa2ac56fecab8153758fd274f0d25ec67d1ee3d5ed8158dbad2fa"
Nov 01 06:37:09.613020 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:37:09.613000    5619 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg_odh-ai-gateway-infra(304878ad-a41a-4349-8ec7-3c7d38f4efbc)\"" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg" podUID="304878ad-a41a-4349-8ec7-3c7d38f4efbc"
Nov 01 06:37:12.806454 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:12.806375    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:37:13.955681 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:13.955651    5619 scope.go:122] "RemoveContainer" containerID="e4acad68fd3fa2ac56fecab8153758fd274f0d25ec67d1ee3d5ed8158dbad2fa"
Nov 01 06:37:14.949797 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:14.949752    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg" event={"ID":"304878ad-a41a-4349-8ec7-3c7d38f4efbc","Type":"ContainerStarted","Data":"335f0346a87a3f515b4a43e81f80035674c8368d764550544b7820995129ad8c"}
Nov 01 06:37:14.950143 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:14.950112    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"
Nov 01 06:37:17.530058 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:17.530014    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:37:20.962958 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:20.962931    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"
Nov 01 06:37:22.320899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:22.320861    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:37:25.879931 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:37:25.879884    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:37:27.050819 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:27.050779    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:37:29.258729 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:29.258689    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:37:31.751160 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:31.751121    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:37:32.509250 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.509215    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"]
Nov 01 06:37:32.509796 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.509782    5619 cpu_manager.go:414] "containerMap: removing container" podUID="c20bff8c-c80c-4776-87d1-7dc271ae1dc7" containerName="payload-processing"
Nov 01 06:37:32.509849 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.509797    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c20bff8c-c80c-4776-87d1-7dc271ae1dc7" containerName="payload-processing"
Nov 01 06:37:32.509849 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.509820    5619 cpu_manager.go:414] "containerMap: removing container" podUID="75319d6b-af9b-4c2f-a53d-71b2a64f3325" containerName="payload-processing"
Nov 01 06:37:32.509849 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.509825    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75319d6b-af9b-4c2f-a53d-71b2a64f3325" containerName="payload-processing"
Nov 01 06:37:32.509849 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.509837    5619 cpu_manager.go:414] "containerMap: removing container" podUID="c86eab09-9ed0-4dc3-9b5e-65b59abef247" containerName="payload-pre-processing"
Nov 01 06:37:32.509849 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.509843    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c86eab09-9ed0-4dc3-9b5e-65b59abef247" containerName="payload-pre-processing"
Nov 01 06:37:32.510009 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.509931    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="c20bff8c-c80c-4776-87d1-7dc271ae1dc7" containerName="payload-processing"
Nov 01 06:37:32.510009 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.509942    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="75319d6b-af9b-4c2f-a53d-71b2a64f3325" containerName="payload-processing"
Nov 01 06:37:32.510009 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.509950    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="c86eab09-9ed0-4dc3-9b5e-65b59abef247" containerName="payload-pre-processing"
Nov 01 06:37:32.513283 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.513266    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.566673 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.566638    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c0662f25-openshift-default-dockercfg-lff96\""
Nov 01 06:37:32.576681 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.576649    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.576821 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.576720    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6vh7\" (UniqueName: \"kubernetes.io/projected/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-kube-api-access-n6vh7\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.576821 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.576757    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.576959 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.576939    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.577058 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.577037    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.577123 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.577109    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.577162 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.577148    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.577200 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.577166    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.577240 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.577201    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.620768 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.620740    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"]
Nov 01 06:37:32.678215 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.678177    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.678419 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.678265    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.678419 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.678301    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.678419 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.678344    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.678419 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.678394    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.678674 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.678446    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.678674 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.678497    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6vh7\" (UniqueName: \"kubernetes.io/projected/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-kube-api-access-n6vh7\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.678674 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.678534    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.678674 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.678664    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.678892 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.678845    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.679043 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.679014    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.679172 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.679016    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.679239 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.679217    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.679337 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.679300    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.680989 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.680967    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.681147 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.681127    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.741383 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.741354    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.741588 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.741569    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6vh7\" (UniqueName: \"kubernetes.io/projected/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-kube-api-access-n6vh7\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-c75dk\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:32.828204 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:32.828127    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:33.255724 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:33.255690    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"]
Nov 01 06:37:33.256834 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:37:33.256802    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6a9d1a_6a42_44d4_a58c_5177b82bb48a.slice/crio-2ee5e265c922401fb1233d1fd5f8ea43ef81573de3ac2afb25255f36ee71ad85 WatchSource:0}: Error finding container 2ee5e265c922401fb1233d1fd5f8ea43ef81573de3ac2afb25255f36ee71ad85: Status 404 returned error can't find the container with id 2ee5e265c922401fb1233d1fd5f8ea43ef81573de3ac2afb25255f36ee71ad85
Nov 01 06:37:33.259442 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:33.259411    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:37:33.259530 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:33.259473    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:37:33.259530 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:33.259502    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:37:33.915020 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:33.914982    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:37:34.079106 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:34.079071    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk" event={"ID":"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a","Type":"ContainerStarted","Data":"92ef82ae79f04d55d350d484d0b2bac03daaac769633945083901b8213f8fb07"}
Nov 01 06:37:34.079379 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:34.079352    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk" event={"ID":"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a","Type":"ContainerStarted","Data":"2ee5e265c922401fb1233d1fd5f8ea43ef81573de3ac2afb25255f36ee71ad85"}
Nov 01 06:37:34.203261 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:34.203147    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk" podStartSLOduration=2.203133321 podStartE2EDuration="2.203133321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:37:34.200650167 +0000 UTC m=+3091.411473676" watchObservedRunningTime="2026-11-01 06:37:34.203133321 +0000 UTC m=+3091.413956829"
Nov 01 06:37:34.828905 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:34.828869    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:34.830738 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:34.830714    5619 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 01 06:37:34.830839 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:34.830771    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk" podUID="1e6a9d1a-6a42-44d4-a58c-5177b82bb48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 01 06:37:35.828536 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:35.828507    5619 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 01 06:37:35.829045 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:35.828555    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk" podUID="1e6a9d1a-6a42-44d4-a58c-5177b82bb48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 01 06:37:36.616835 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:36.616800    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:37:36.829424 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:36.829393    5619 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 01 06:37:36.829787 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:36.829479    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk" podUID="1e6a9d1a-6a42-44d4-a58c-5177b82bb48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 01 06:37:37.829021 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:37.828987    5619 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 01 06:37:37.829206 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:37.829064    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk" podUID="1e6a9d1a-6a42-44d4-a58c-5177b82bb48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 01 06:37:38.828582 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:38.828548    5619 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 01 06:37:38.828955 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:38.828597    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk" podUID="1e6a9d1a-6a42-44d4-a58c-5177b82bb48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 01 06:37:39.833739 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:39.833704    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:39.834134 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:39.834109    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:39.834928 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:39.834908    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:37:40.715822 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:40.715789    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"]
Nov 01 06:37:40.716066 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:40.716025    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h" podUID="6580ffaf-1ef0-47e4-b1b7-aa722e3117f2" containerName="maas-api" containerID="cri-o://5b525a79dacc17ad552ee03a9c483f8a2b8c6766ae755577c5dd217295532c10" gracePeriod=30
Nov 01 06:37:40.814597 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:40.814555    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:37:41.010127 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:41.010098    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"
Nov 01 06:37:41.101124 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:41.101089    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-metrics-certs\") pod \"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2\" (UID: \"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2\") "
Nov 01 06:37:41.101318 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:41.101277    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-kube-api-access-qsspm\" (UniqueName: \"kubernetes.io/projected/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-kube-api-access-qsspm\") pod \"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2\" (UID: \"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2\") "
Nov 01 06:37:41.101429 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:41.101359    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-maas-api-tls\") pod \"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2\" (UID: \"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2\") "
Nov 01 06:37:41.101483 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:41.101436    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-openshift-service-ca\") pod \"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2\" (UID: \"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2\") "
Nov 01 06:37:41.102032 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:41.102000    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-openshift-service-ca" pod "6580ffaf-1ef0-47e4-b1b7-aa722e3117f2" (UID: "6580ffaf-1ef0-47e4-b1b7-aa722e3117f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:37:41.103732 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:41.103706    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-kube-api-access-qsspm" pod "6580ffaf-1ef0-47e4-b1b7-aa722e3117f2" (UID: "6580ffaf-1ef0-47e4-b1b7-aa722e3117f2"). InnerVolumeSpecName "kube-api-access-qsspm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:37:41.104072 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:41.104047    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-maas-api-tls" pod "6580ffaf-1ef0-47e4-b1b7-aa722e3117f2" (UID: "6580ffaf-1ef0-47e4-b1b7-aa722e3117f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:37:41.104208 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:41.104194    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-metrics-certs" pod "6580ffaf-1ef0-47e4-b1b7-aa722e3117f2" (UID: "6580ffaf-1ef0-47e4-b1b7-aa722e3117f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:37:41.128350 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:41.128019    5619 generic.go:356] "Generic (PLEG): container finished" podID="6580ffaf-1ef0-47e4-b1b7-aa722e3117f2" containerID="5b525a79dacc17ad552ee03a9c483f8a2b8c6766ae755577c5dd217295532c10" exitCode=0
Nov 01 06:37:41.129433 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:41.129127    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"
Nov 01 06:37:41.135354 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:41.133098    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h" event={"ID":"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2","Type":"ContainerDied","Data":"5b525a79dacc17ad552ee03a9c483f8a2b8c6766ae755577c5dd217295532c10"}
Nov 01 06:37:41.135354 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:41.133217    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h" event={"ID":"6580ffaf-1ef0-47e4-b1b7-aa722e3117f2","Type":"ContainerDied","Data":"e5bdcf4302a150b65b5861a635b0294f82de2a6c2dea07db152686df916750d2"}
Nov 01 06:37:41.135354 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:41.133290    5619 scope.go:122] "RemoveContainer" containerID="5b525a79dacc17ad552ee03a9c483f8a2b8c6766ae755577c5dd217295532c10"
Nov 01 06:37:41.177433 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:41.177308    5619 scope.go:122] "RemoveContainer" containerID="5b525a79dacc17ad552ee03a9c483f8a2b8c6766ae755577c5dd217295532c10"
Nov 01 06:37:41.177765 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:37:41.177737    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b525a79dacc17ad552ee03a9c483f8a2b8c6766ae755577c5dd217295532c10\": container with ID starting with 5b525a79dacc17ad552ee03a9c483f8a2b8c6766ae755577c5dd217295532c10 not found: ID does not exist" containerID="5b525a79dacc17ad552ee03a9c483f8a2b8c6766ae755577c5dd217295532c10"
Nov 01 06:37:41.177865 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:41.177781    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b525a79dacc17ad552ee03a9c483f8a2b8c6766ae755577c5dd217295532c10"} err="failed to get container status \"5b525a79dacc17ad552ee03a9c483f8a2b8c6766ae755577c5dd217295532c10\": rpc error: code = NotFound desc = could not find container \"5b525a79dacc17ad552ee03a9c483f8a2b8c6766ae755577c5dd217295532c10\": container with ID starting with 5b525a79dacc17ad552ee03a9c483f8a2b8c6766ae755577c5dd217295532c10 not found: ID does not exist"
Nov 01 06:37:41.203002 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:41.202973    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsspm\" (UniqueName: \"kubernetes.io/projected/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-kube-api-access-qsspm\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:41.203203 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:41.203185    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:41.203272 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:41.203253    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:41.203317 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:41.203272    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:37:41.302475 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:41.302373    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"]
Nov 01 06:37:41.304914 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:41.304878    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3680c5-7d84756df5-spl9h"]
Nov 01 06:37:42.539874 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:42.539840    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:37:42.964358 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:42.964307    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6580ffaf-1ef0-47e4-b1b7-aa722e3117f2" path="/var/lib/kubelet/pods/6580ffaf-1ef0-47e4-b1b7-aa722e3117f2/volumes"
Nov 01 06:37:47.126675 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.126633    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:37:47.340063 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.340027    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:37:47.487455 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.487368    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:37:47.633199 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.633164    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"]
Nov 01 06:37:47.633760 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.633744    5619 cpu_manager.go:414] "containerMap: removing container" podUID="6580ffaf-1ef0-47e4-b1b7-aa722e3117f2" containerName="maas-api"
Nov 01 06:37:47.633760 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.633760    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6580ffaf-1ef0-47e4-b1b7-aa722e3117f2" containerName="maas-api"
Nov 01 06:37:47.633905 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.633845    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="6580ffaf-1ef0-47e4-b1b7-aa722e3117f2" containerName="maas-api"
Nov 01 06:37:47.638441 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.638421    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.684420 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.684382    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"]
Nov 01 06:37:47.688925 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.688896    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.700020 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.699991    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.700153 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.700030    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.700153 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.700060    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.700228 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.700181    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.700228 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.700216    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqpzm\" (UniqueName: \"kubernetes.io/projected/29d290fb-bc3b-4b96-849c-882ab07bf143-kube-api-access-wqpzm\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.700300 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.700261    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqnmm\" (UniqueName: \"kubernetes.io/projected/b92297e9-8458-4787-96f6-e9025d2a6a31-kube-api-access-pqnmm\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.700300 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.700286    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29d290fb-bc3b-4b96-849c-882ab07bf143-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.700426 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.700307    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.700426 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.700422    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.700505 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.700496    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.700589 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.700568    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.700653 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.700625    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.700712 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.700672    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.700712 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.700699    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92297e9-8458-4787-96f6-e9025d2a6a31-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.700810 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.700740    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.700852 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.700811    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.700916 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.700856    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.700966 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.700924    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.741002 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.740920    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"]
Nov 01 06:37:47.794402 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.794372    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"]
Nov 01 06:37:47.801936 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.801906    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29d290fb-bc3b-4b96-849c-882ab07bf143-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.802126 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.801943    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.802126 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.801974    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.802126 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.802080    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.802126 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.802103    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.802403 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.802151    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.802403 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.802189    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.802403 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.802216    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92297e9-8458-4787-96f6-e9025d2a6a31-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.802403 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.802257    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.802403 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.802280    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.802403 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.802313    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.802403 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.802385    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.802783 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.802447    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.802783 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.802482    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.802783 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.802522    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.802783 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.802555    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.802783 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.802598    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqpzm\" (UniqueName: \"kubernetes.io/projected/29d290fb-bc3b-4b96-849c-882ab07bf143-kube-api-access-wqpzm\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.802783 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.802670    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqnmm\" (UniqueName: \"kubernetes.io/projected/b92297e9-8458-4787-96f6-e9025d2a6a31-kube-api-access-pqnmm\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.802783 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.802729    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29d290fb-bc3b-4b96-849c-882ab07bf143-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.803133 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.803046    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.803133 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.803082    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.803250 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.803230    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.804873 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.803707    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-data\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.804873 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.804636    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92297e9-8458-4787-96f6-e9025d2a6a31-istiod-ca-cert\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.805396 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.805370    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.805505 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.805430    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.805974 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.805947    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-workload-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.806071 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.806031    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-credential-socket\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.806071 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.806060    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-workload-certs\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.806306 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.806278    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.807624 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.807598    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-envoy\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.808502 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.808478    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-podinfo\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.863541 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.863515    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.864107 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.864089    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-token\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.864524 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.864502    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqpzm\" (UniqueName: \"kubernetes.io/projected/29d290fb-bc3b-4b96-849c-882ab07bf143-kube-api-access-wqpzm\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:47.870020 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.869995    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqnmm\" (UniqueName: \"kubernetes.io/projected/b92297e9-8458-4787-96f6-e9025d2a6a31-kube-api-access-pqnmm\") pod \"e2e-mt-c0662f25-openshift-default-86d794b649-vvd94\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") " pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:47.952971 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:47.952935    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:48.005498 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:48.005390    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:48.391459 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:48.391428    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"]
Nov 01 06:37:48.391818 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:37:48.391551    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92297e9_8458_4787_96f6_e9025d2a6a31.slice/crio-9286f22540c5308189585032fc2c346e30093275ec76a02f65ec4d67ae67a378 WatchSource:0}: Error finding container 9286f22540c5308189585032fc2c346e30093275ec76a02f65ec4d67ae67a378: Status 404 returned error can't find the container with id 9286f22540c5308189585032fc2c346e30093275ec76a02f65ec4d67ae67a378
Nov 01 06:37:48.394492 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:48.394457    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:37:48.394615 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:48.394554    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:37:48.394615 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:48.394597    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:37:48.438059 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:48.438034    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"]
Nov 01 06:37:48.451704 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:37:48.451676    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d290fb_bc3b_4b96_849c_882ab07bf143.slice/crio-5d95899bd972159c8bf0f5159fed817c3a853b7c17c85969c9f62f9a63800558 WatchSource:0}: Error finding container 5d95899bd972159c8bf0f5159fed817c3a853b7c17c85969c9f62f9a63800558: Status 404 returned error can't find the container with id 5d95899bd972159c8bf0f5159fed817c3a853b7c17c85969c9f62f9a63800558
Nov 01 06:37:48.454757 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:48.454726    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:37:48.454949 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:48.454791    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:37:48.454949 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:48.454820    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:37:49.202347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:49.200588    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7" event={"ID":"29d290fb-bc3b-4b96-849c-882ab07bf143","Type":"ContainerStarted","Data":"873a3cbebf6832b63c7f158a602e0bcd85ea76e8bed1ee9dc707f0700a364191"}
Nov 01 06:37:49.202347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:49.200997    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7" event={"ID":"29d290fb-bc3b-4b96-849c-882ab07bf143","Type":"ContainerStarted","Data":"5d95899bd972159c8bf0f5159fed817c3a853b7c17c85969c9f62f9a63800558"}
Nov 01 06:37:49.202937 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:49.202910    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94" event={"ID":"b92297e9-8458-4787-96f6-e9025d2a6a31","Type":"ContainerStarted","Data":"28f4379fbd0c224c4de13336cdc30f2aabf545e20243d1deddd8b775c277f852"}
Nov 01 06:37:49.202984 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:49.202964    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94" event={"ID":"b92297e9-8458-4787-96f6-e9025d2a6a31","Type":"ContainerStarted","Data":"9286f22540c5308189585032fc2c346e30093275ec76a02f65ec4d67ae67a378"}
Nov 01 06:37:49.339037 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:49.338976    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7" podStartSLOduration=2.338956803 podStartE2EDuration="2.338956803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:37:49.337222757 +0000 UTC m=+3106.548046267" watchObservedRunningTime="2026-11-01 06:37:49.338956803 +0000 UTC m=+3106.549780314"
Nov 01 06:37:49.957345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:49.955759    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:49.963346 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:49.961445    5619 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 06:37:49.963346 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:49.961511    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94" podUID="b92297e9-8458-4787-96f6-e9025d2a6a31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 06:37:50.008435 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:50.008403    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:50.008779 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:50.008755    5619 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 01 06:37:50.008963 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:50.008937    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7" podUID="29d290fb-bc3b-4b96-849c-882ab07bf143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 01 06:37:50.954346 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:50.953745    5619 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 06:37:50.954346 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:50.953808    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94" podUID="b92297e9-8458-4787-96f6-e9025d2a6a31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 06:37:51.019448 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:51.019409    5619 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 01 06:37:51.019940 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:51.019480    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7" podUID="29d290fb-bc3b-4b96-849c-882ab07bf143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 01 06:37:51.027238 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:51.027200    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:37:51.662170 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:51.662132    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:37:51.968400 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:51.965639    5619 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 06:37:51.968400 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:51.965710    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94" podUID="b92297e9-8458-4787-96f6-e9025d2a6a31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 06:37:52.006691 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:52.006645    5619 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 01 06:37:52.006898 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:52.006700    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7" podUID="29d290fb-bc3b-4b96-849c-882ab07bf143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 01 06:37:52.275987 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:52.275901    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:37:52.541336 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:52.541226    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:37:52.905372 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:52.905002    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:37:52.977347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:52.976658    5619 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 06:37:52.977347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:52.976716    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94" podUID="b92297e9-8458-4787-96f6-e9025d2a6a31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 06:37:53.015372 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:53.006505    5619 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 01 06:37:53.015372 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:53.006603    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7" podUID="29d290fb-bc3b-4b96-849c-882ab07bf143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 01 06:37:53.967346 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:53.966630    5619 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 06:37:53.967346 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:53.966702    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94" podUID="b92297e9-8458-4787-96f6-e9025d2a6a31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 06:37:54.009771 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:54.009639    5619 patch_prober.go:29] interesting pod/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 01 06:37:54.009771 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:54.009691    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7" podUID="29d290fb-bc3b-4b96-849c-882ab07bf143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 01 06:37:55.012975 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:55.012746    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:55.013494 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:55.013105    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:55.014749 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:55.014726    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:37:55.024936 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:55.024910    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:55.025560 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:55.025526    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:55.026936 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:55.026913    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:37:55.188128 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:55.188075    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94" podStartSLOduration=8.188060028 podStartE2EDuration="8.188060028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:37:49.488631091 +0000 UTC m=+3106.699454601" watchObservedRunningTime="2026-11-01 06:37:55.188060028 +0000 UTC m=+3112.398883536"
Nov 01 06:37:55.886195 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:37:55.886155    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:37:57.865302 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:37:57.865257    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:02.580726 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:02.580675    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:02.784766 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:02.784730    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"]
Nov 01 06:38:02.788804 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:02.788781    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:02.843176 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:02.843135    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-c673e5-openshift-default-dockercfg-kstbj\""
Nov 01 06:38:02.899935 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:02.899896    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:02.912310 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:02.912276    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"]
Nov 01 06:38:02.943079 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:02.943031    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-data\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:02.943282 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:02.943114    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-workload-certs\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:02.943282 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:02.943144    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-envoy\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:02.943282 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:02.943244    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-token\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:02.943488 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:02.943338    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-workload-socket\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:02.943488 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:02.943418    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xpcw\" (UniqueName: \"kubernetes.io/projected/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-kube-api-access-6xpcw\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:02.943488 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:02.943445    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-podinfo\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:02.943662 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:02.943486    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istiod-ca-cert\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:02.943662 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:02.943532    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-credential-socket\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:03.044903 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.044860    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-workload-socket\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:03.045219 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.045199    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xpcw\" (UniqueName: \"kubernetes.io/projected/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-kube-api-access-6xpcw\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:03.045387 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.045357    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-workload-socket\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:03.045525 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.045367    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-podinfo\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:03.045525 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.045455    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istiod-ca-cert\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:03.045525 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.045500    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-credential-socket\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:03.045704 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.045571    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-data\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:03.045704 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.045649    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-workload-certs\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:03.045704 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.045681    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-envoy\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:03.045948 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.045826    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-token\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:03.046246 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.046222    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-data\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:03.046528 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.046504    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-workload-certs\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:03.047052 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.047006    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istiod-ca-cert\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:03.047306 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.047259    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-credential-socket\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:03.049228 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.049205    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-envoy\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:03.049348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.049312    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-podinfo\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:03.111598 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.111511    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-token\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:03.111764 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.111653    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xpcw\" (UniqueName: \"kubernetes.io/projected/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-kube-api-access-6xpcw\") pod \"e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") " pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:03.456268 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.456191    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-c673e5-openshift-default-dockercfg-kstbj\""
Nov 01 06:38:03.464246 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.464212    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:03.480893 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.480855    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:03.895874 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.895847    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"]
Nov 01 06:38:03.896280 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:38:03.896256    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff3bbf5_47d7_4a26_95c6_ed22acc794c3.slice/crio-14a5f5cf295177838881db81eaf77958fb0c2db24b0b8a051719c8b9ffaa3811 WatchSource:0}: Error finding container 14a5f5cf295177838881db81eaf77958fb0c2db24b0b8a051719c8b9ffaa3811: Status 404 returned error can't find the container with id 14a5f5cf295177838881db81eaf77958fb0c2db24b0b8a051719c8b9ffaa3811
Nov 01 06:38:03.898739 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.898709    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:38:03.898807 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.898773    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:38:03.898807 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:03.898802    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:38:04.300831 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:04.300793    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6" event={"ID":"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3","Type":"ContainerStarted","Data":"ff1aca6046adac53531db1e0751b16de9bba8f3bffcb5895fa79c9b096cf703c"}
Nov 01 06:38:04.301012 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:04.300851    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6" event={"ID":"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3","Type":"ContainerStarted","Data":"14a5f5cf295177838881db81eaf77958fb0c2db24b0b8a051719c8b9ffaa3811"}
Nov 01 06:38:04.464480 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:04.464447    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:04.470802 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:04.470771    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:04.529429 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:04.529380    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6" podStartSLOduration=2.529364171 podStartE2EDuration="2.529364171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:38:04.476849806 +0000 UTC m=+3121.687673297" watchObservedRunningTime="2026-11-01 06:38:04.529364171 +0000 UTC m=+3121.740187680"
Nov 01 06:38:05.306562 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:05.306527    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:05.307571 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:05.307551    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:38:07.269705 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:07.269658    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:07.460528 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:07.460495    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:07.602927 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:07.602869    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:07.834335 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:07.834302    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:08.942757 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:08.942722    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:09.135499 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:09.135462    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:10.498723 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:10.498686    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:10.676909 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:10.676868    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:10.913527 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:10.913497    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g"]
Nov 01 06:38:10.923815 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:10.923780    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g"
Nov 01 06:38:10.975141 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:10.975112    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g"]
Nov 01 06:38:11.037369 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:11.035658    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-c673e5-dockercfg-cxdkj\""
Nov 01 06:38:11.037369 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:11.035771    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-c673e5\""
Nov 01 06:38:11.065038 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:11.065005    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e533b8-8047-477b-a358-c72cb0b096a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g\" (UID: \"c1e533b8-8047-477b-a358-c72cb0b096a1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g"
Nov 01 06:38:11.065253 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:11.065141    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrdrk\" (UniqueName: \"kubernetes.io/projected/c1e533b8-8047-477b-a358-c72cb0b096a1-kube-api-access-wrdrk\") pod \"payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g\" (UID: \"c1e533b8-8047-477b-a358-c72cb0b096a1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g"
Nov 01 06:38:11.166384 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:11.166244    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e533b8-8047-477b-a358-c72cb0b096a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g\" (UID: \"c1e533b8-8047-477b-a358-c72cb0b096a1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g"
Nov 01 06:38:11.166578 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:11.166392    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrdrk\" (UniqueName: \"kubernetes.io/projected/c1e533b8-8047-477b-a358-c72cb0b096a1-kube-api-access-wrdrk\") pod \"payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g\" (UID: \"c1e533b8-8047-477b-a358-c72cb0b096a1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g"
Nov 01 06:38:11.167736 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:11.167707    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e533b8-8047-477b-a358-c72cb0b096a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g\" (UID: \"c1e533b8-8047-477b-a358-c72cb0b096a1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g"
Nov 01 06:38:11.231216 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:11.231189    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrdrk\" (UniqueName: \"kubernetes.io/projected/c1e533b8-8047-477b-a358-c72cb0b096a1-kube-api-access-wrdrk\") pod \"payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g\" (UID: \"c1e533b8-8047-477b-a358-c72cb0b096a1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g"
Nov 01 06:38:11.238009 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:11.237981    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g"
Nov 01 06:38:11.615830 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:11.615803    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g"]
Nov 01 06:38:11.617087 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:38:11.617063    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e533b8_8047_477b_a358_c72cb0b096a1.slice/crio-39f72cf319bed752264f8cbd14c1d9c3560f86c276ddf8288f30c16b59bead9c WatchSource:0}: Error finding container 39f72cf319bed752264f8cbd14c1d9c3560f86c276ddf8288f30c16b59bead9c: Status 404 returned error can't find the container with id 39f72cf319bed752264f8cbd14c1d9c3560f86c276ddf8288f30c16b59bead9c
Nov 01 06:38:12.356342 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:12.356292    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g" event={"ID":"c1e533b8-8047-477b-a358-c72cb0b096a1","Type":"ContainerStarted","Data":"f9c54eb9d92b40fd962fd9392006d24bbee4104bf81df486d438d15612901514"}
Nov 01 06:38:12.356517 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:12.356381    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g" event={"ID":"c1e533b8-8047-477b-a358-c72cb0b096a1","Type":"ContainerStarted","Data":"39f72cf319bed752264f8cbd14c1d9c3560f86c276ddf8288f30c16b59bead9c"}
Nov 01 06:38:12.447389 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:12.447354    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:12.474179 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:12.474130    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g" podStartSLOduration=2.474106745 podStartE2EDuration="2.474106745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:38:12.473219042 +0000 UTC m=+3129.684042551" watchObservedRunningTime="2026-11-01 06:38:12.474106745 +0000 UTC m=+3129.684930257"
Nov 01 06:38:12.880370 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:12.880310    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:13.362529 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:13.362500    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g"
Nov 01 06:38:17.204821 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:17.204780    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:21.902195 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:21.902157    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:24.371988 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:24.371959    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g"
Nov 01 06:38:25.880029 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:38:25.879987    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:38:26.590235 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:26.590199    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:31.163863 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:31.163827    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:31.359591 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:31.359558    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:34.259057 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:34.259013    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:36.156908 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:36.156872    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:40.907587 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:40.907500    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:44.066296 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:44.066260    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:44.899249 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:44.899210    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:45.656734 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:45.656700    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:49.594299 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:49.594265    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:50.324946 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:50.324908    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:50.513447 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:50.513403    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:51.192335 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:51.192289    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:51.403699 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:51.403666    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:55.473479 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:55.473432    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:38:55.880448 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:38:55.880409    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:38:56.099873 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:38:56.099829    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:00.187734 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:00.187698    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:00.377908 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:00.377866    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:04.992606 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:04.992567    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:09.601688 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:09.601624    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:09.788108 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:09.788066    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:09.991133 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:09.991028    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:13.848356 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:13.848291    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:13.989247 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:13.989207    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:14.418722 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:14.418680    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:14.775457 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:14.775369    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:19.598093 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:19.598061    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:24.258886 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:24.258848    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:25.881801 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:39:25.881752    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:39:29.058478 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:29.058442    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:29.947537 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:29.947487    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:33.767428 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:33.767384    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:34.610509 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:34.610475    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:39.093372 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:39.091775    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:43.282557 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:43.282487    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:44.098960 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:44.098923    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:47.176230 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:47.176150    5619 generic.go:356] "Generic (PLEG): container finished" podID="9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" containerID="ff1aca6046adac53531db1e0751b16de9bba8f3bffcb5895fa79c9b096cf703c" exitCode=0
Nov 01 06:39:47.176230 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:47.176218    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6" event={"ID":"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3","Type":"ContainerDied","Data":"ff1aca6046adac53531db1e0751b16de9bba8f3bffcb5895fa79c9b096cf703c"}
Nov 01 06:39:47.176710 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:47.176652    5619 scope.go:122] "RemoveContainer" containerID="ff1aca6046adac53531db1e0751b16de9bba8f3bffcb5895fa79c9b096cf703c"
Nov 01 06:39:47.178552 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:47.178519    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:39:47.178645 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:47.178624    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:39:47.178707 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:47.178678    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:39:48.184047 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:48.184014    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6" event={"ID":"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3","Type":"ContainerStarted","Data":"349417e0f0d0fc1c8a41c5bd39281efc96e5db156544cf2df62191d451fa514a"}
Nov 01 06:39:48.465691 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:48.465607    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:39:48.475495 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:48.475446    5619 patch_prober.go:29] interesting pod/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 01 06:39:48.475774 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:48.475736    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6" podUID="9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 01 06:39:48.929389 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:48.929355    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:49.471355 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:49.467468    5619 patch_prober.go:29] interesting pod/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 01 06:39:49.471355 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:49.467558    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6" podUID="9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 01 06:39:50.307438 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:50.307401    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:39:50.464689 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:50.464650    5619 patch_prober.go:29] interesting pod/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 01 06:39:50.464981 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:50.464949    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6" podUID="9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 01 06:39:51.468346 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:51.467650    5619 patch_prober.go:29] interesting pod/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 01 06:39:51.468346 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:51.467719    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6" podUID="9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 01 06:39:52.474706 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:52.474671    5619 patch_prober.go:29] interesting pod/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 01 06:39:52.475288 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:52.475259    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6" podUID="9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 01 06:39:53.047274 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:53.047243    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"]
Nov 01 06:39:53.047521 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:53.047496    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg" podUID="304878ad-a41a-4349-8ec7-3c7d38f4efbc" containerName="maas-api" containerID="cri-o://335f0346a87a3f515b4a43e81f80035674c8368d764550544b7820995129ad8c" gracePeriod=30
Nov 01 06:39:53.153505 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:53.153463    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:53.227348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:53.227290    5619 generic.go:356] "Generic (PLEG): container finished" podID="304878ad-a41a-4349-8ec7-3c7d38f4efbc" containerID="335f0346a87a3f515b4a43e81f80035674c8368d764550544b7820995129ad8c" exitCode=0
Nov 01 06:39:53.227502 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:53.227481    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg" event={"ID":"304878ad-a41a-4349-8ec7-3c7d38f4efbc","Type":"ContainerDied","Data":"335f0346a87a3f515b4a43e81f80035674c8368d764550544b7820995129ad8c"}
Nov 01 06:39:53.227575 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:53.227544    5619 scope.go:122] "RemoveContainer" containerID="e4acad68fd3fa2ac56fecab8153758fd274f0d25ec67d1ee3d5ed8158dbad2fa"
Nov 01 06:39:53.374679 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:53.374639    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"
Nov 01 06:39:53.440727 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:53.440692    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/304878ad-a41a-4349-8ec7-3c7d38f4efbc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/304878ad-a41a-4349-8ec7-3c7d38f4efbc-metrics-certs\") pod \"304878ad-a41a-4349-8ec7-3c7d38f4efbc\" (UID: \"304878ad-a41a-4349-8ec7-3c7d38f4efbc\") "
Nov 01 06:39:53.440936 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:53.440749    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/304878ad-a41a-4349-8ec7-3c7d38f4efbc-kube-api-access-rvf6c\" (UniqueName: \"kubernetes.io/projected/304878ad-a41a-4349-8ec7-3c7d38f4efbc-kube-api-access-rvf6c\") pod \"304878ad-a41a-4349-8ec7-3c7d38f4efbc\" (UID: \"304878ad-a41a-4349-8ec7-3c7d38f4efbc\") "
Nov 01 06:39:53.440936 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:53.440828    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/304878ad-a41a-4349-8ec7-3c7d38f4efbc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/304878ad-a41a-4349-8ec7-3c7d38f4efbc-maas-api-tls\") pod \"304878ad-a41a-4349-8ec7-3c7d38f4efbc\" (UID: \"304878ad-a41a-4349-8ec7-3c7d38f4efbc\") "
Nov 01 06:39:53.441064 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:53.440956    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/304878ad-a41a-4349-8ec7-3c7d38f4efbc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/304878ad-a41a-4349-8ec7-3c7d38f4efbc-openshift-service-ca\") pod \"304878ad-a41a-4349-8ec7-3c7d38f4efbc\" (UID: \"304878ad-a41a-4349-8ec7-3c7d38f4efbc\") "
Nov 01 06:39:53.441845 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:53.441790    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/304878ad-a41a-4349-8ec7-3c7d38f4efbc-openshift-service-ca" pod "304878ad-a41a-4349-8ec7-3c7d38f4efbc" (UID: "304878ad-a41a-4349-8ec7-3c7d38f4efbc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:39:53.443800 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:53.443771    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/304878ad-a41a-4349-8ec7-3c7d38f4efbc-maas-api-tls" pod "304878ad-a41a-4349-8ec7-3c7d38f4efbc" (UID: "304878ad-a41a-4349-8ec7-3c7d38f4efbc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:39:53.443917 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:53.443821    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/304878ad-a41a-4349-8ec7-3c7d38f4efbc-kube-api-access-rvf6c" pod "304878ad-a41a-4349-8ec7-3c7d38f4efbc" (UID: "304878ad-a41a-4349-8ec7-3c7d38f4efbc"). InnerVolumeSpecName "kube-api-access-rvf6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:39:53.445891 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:53.445859    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/304878ad-a41a-4349-8ec7-3c7d38f4efbc-metrics-certs" pod "304878ad-a41a-4349-8ec7-3c7d38f4efbc" (UID: "304878ad-a41a-4349-8ec7-3c7d38f4efbc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:39:53.478398 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:53.478360    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:39:53.480999 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:53.480974    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:39:53.542481 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:53.542443    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/304878ad-a41a-4349-8ec7-3c7d38f4efbc-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:39:53.542481 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:53.542480    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvf6c\" (UniqueName: \"kubernetes.io/projected/304878ad-a41a-4349-8ec7-3c7d38f4efbc-kube-api-access-rvf6c\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:39:53.542740 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:53.542497    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/304878ad-a41a-4349-8ec7-3c7d38f4efbc-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:39:53.542740 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:53.542513    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/304878ad-a41a-4349-8ec7-3c7d38f4efbc-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:39:54.236009 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:54.235973    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"
Nov 01 06:39:54.236009 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:54.235993    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg" event={"ID":"304878ad-a41a-4349-8ec7-3c7d38f4efbc","Type":"ContainerDied","Data":"2b9a8deaf5fcb43a777d31d9dd5efbe71194a2b8128421fdfd629a1c3523adac"}
Nov 01 06:39:54.236240 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:54.236098    5619 scope.go:122] "RemoveContainer" containerID="335f0346a87a3f515b4a43e81f80035674c8368d764550544b7820995129ad8c"
Nov 01 06:39:54.405201 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:54.405166    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"]
Nov 01 06:39:54.408641 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:54.408611    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0662f25-78f9c746dd-pzhvg"]
Nov 01 06:39:54.825790 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:54.825747    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:54.966538 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:54.966505    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="304878ad-a41a-4349-8ec7-3c7d38f4efbc" path="/var/lib/kubelet/pods/304878ad-a41a-4349-8ec7-3c7d38f4efbc/volumes"
Nov 01 06:39:54.976656 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:54.976624    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:55.427690 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:55.427643    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:55.430881 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:55.430852    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"]
Nov 01 06:39:55.431126 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:55.431100    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk" podUID="1e6a9d1a-6a42-44d4-a58c-5177b82bb48a" containerName="istio-proxy" containerID="cri-o://92ef82ae79f04d55d350d484d0b2bac03daaac769633945083901b8213f8fb07" gracePeriod=30
Nov 01 06:39:55.443264 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:55.443225    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"]
Nov 01 06:39:55.443500 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:55.443478    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94" podUID="b92297e9-8458-4787-96f6-e9025d2a6a31" containerName="istio-proxy" containerID="cri-o://28f4379fbd0c224c4de13336cdc30f2aabf545e20243d1deddd8b775c277f852" gracePeriod=30
Nov 01 06:39:55.467294 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:55.467266    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"]
Nov 01 06:39:55.467549 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:55.467524    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7" podUID="29d290fb-bc3b-4b96-849c-882ab07bf143" containerName="istio-proxy" containerID="cri-o://873a3cbebf6832b63c7f158a602e0bcd85ea76e8bed1ee9dc707f0700a364191" gracePeriod=30
Nov 01 06:39:55.881061 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:39:55.881019    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:39:55.909879 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:55.909842    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:56.048906 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:56.048873    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:39:59.964212 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:39:59.964142    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:00.755520 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.755486    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:40:00.818764 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.818737    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:40:00.822401 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.822380    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:40:00.840862 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.840841    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b92297e9-8458-4787-96f6-e9025d2a6a31-kube-api-access-pqnmm\" (UniqueName: \"kubernetes.io/projected/b92297e9-8458-4787-96f6-e9025d2a6a31-kube-api-access-pqnmm\") pod \"b92297e9-8458-4787-96f6-e9025d2a6a31\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") "
Nov 01 06:40:00.840969 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.840882    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-data\") pod \"b92297e9-8458-4787-96f6-e9025d2a6a31\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") "
Nov 01 06:40:00.840969 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.840925    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-podinfo\") pod \"b92297e9-8458-4787-96f6-e9025d2a6a31\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") "
Nov 01 06:40:00.840969 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.840966    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-workload-socket\") pod \"b92297e9-8458-4787-96f6-e9025d2a6a31\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") "
Nov 01 06:40:00.841132 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.841026    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b92297e9-8458-4787-96f6-e9025d2a6a31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92297e9-8458-4787-96f6-e9025d2a6a31-istiod-ca-cert\") pod \"b92297e9-8458-4787-96f6-e9025d2a6a31\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") "
Nov 01 06:40:00.841132 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.841058    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-credential-socket\") pod \"b92297e9-8458-4787-96f6-e9025d2a6a31\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") "
Nov 01 06:40:00.841243 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.841165    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-workload-certs\") pod \"b92297e9-8458-4787-96f6-e9025d2a6a31\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") "
Nov 01 06:40:00.841304 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.841238    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-token\" (UniqueName: \"kubernetes.io/projected/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-token\") pod \"b92297e9-8458-4787-96f6-e9025d2a6a31\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") "
Nov 01 06:40:00.841304 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.841275    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-data" pod "b92297e9-8458-4787-96f6-e9025d2a6a31" (UID: "b92297e9-8458-4787-96f6-e9025d2a6a31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.841304 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.841287    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-envoy\") pod \"b92297e9-8458-4787-96f6-e9025d2a6a31\" (UID: \"b92297e9-8458-4787-96f6-e9025d2a6a31\") "
Nov 01 06:40:00.841304 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.841287    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-workload-socket" pod "b92297e9-8458-4787-96f6-e9025d2a6a31" (UID: "b92297e9-8458-4787-96f6-e9025d2a6a31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.841564 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.841481    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-workload-certs" pod "b92297e9-8458-4787-96f6-e9025d2a6a31" (UID: "b92297e9-8458-4787-96f6-e9025d2a6a31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.841564 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.841537    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b92297e9-8458-4787-96f6-e9025d2a6a31-istiod-ca-cert" pod "b92297e9-8458-4787-96f6-e9025d2a6a31" (UID: "b92297e9-8458-4787-96f6-e9025d2a6a31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:40:00.841849 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.841823    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-credential-socket" pod "b92297e9-8458-4787-96f6-e9025d2a6a31" (UID: "b92297e9-8458-4787-96f6-e9025d2a6a31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.841960 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.841916    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.841960 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.841936    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.842092 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.841960    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.842092 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.841972    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92297e9-8458-4787-96f6-e9025d2a6a31-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.842092 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.841982    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.843112 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.843092    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92297e9-8458-4787-96f6-e9025d2a6a31-kube-api-access-pqnmm" pod "b92297e9-8458-4787-96f6-e9025d2a6a31" (UID: "b92297e9-8458-4787-96f6-e9025d2a6a31"). InnerVolumeSpecName "kube-api-access-pqnmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:00.843467 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.843446    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-token" pod "b92297e9-8458-4787-96f6-e9025d2a6a31" (UID: "b92297e9-8458-4787-96f6-e9025d2a6a31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:00.843628 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.843601    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-envoy" pod "b92297e9-8458-4787-96f6-e9025d2a6a31" (UID: "b92297e9-8458-4787-96f6-e9025d2a6a31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.843628 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.843608    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-podinfo" pod "b92297e9-8458-4787-96f6-e9025d2a6a31" (UID: "b92297e9-8458-4787-96f6-e9025d2a6a31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:40:00.942917 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.942875    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-podinfo\") pod \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") "
Nov 01 06:40:00.943084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.942945    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-credential-socket\") pod \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") "
Nov 01 06:40:00.943084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.943014    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-token\" (UniqueName: \"kubernetes.io/projected/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-token\") pod \"29d290fb-bc3b-4b96-849c-882ab07bf143\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") "
Nov 01 06:40:00.943084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.943061    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-workload-socket\") pod \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") "
Nov 01 06:40:00.943292 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.943087    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-workload-certs\") pod \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") "
Nov 01 06:40:00.943292 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.943154    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-workload-socket\") pod \"29d290fb-bc3b-4b96-849c-882ab07bf143\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") "
Nov 01 06:40:00.943292 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.943186    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-envoy\") pod \"29d290fb-bc3b-4b96-849c-882ab07bf143\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") "
Nov 01 06:40:00.943292 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.943227    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-data\") pod \"29d290fb-bc3b-4b96-849c-882ab07bf143\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") "
Nov 01 06:40:00.943292 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.943262    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-envoy\") pod \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") "
Nov 01 06:40:00.943611 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.943309    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-credential-socket" pod "1e6a9d1a-6a42-44d4-a58c-5177b82bb48a" (UID: "1e6a9d1a-6a42-44d4-a58c-5177b82bb48a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.943611 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.943369    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-credential-socket\") pod \"29d290fb-bc3b-4b96-849c-882ab07bf143\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") "
Nov 01 06:40:00.943611 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.943548    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-workload-certs" pod "1e6a9d1a-6a42-44d4-a58c-5177b82bb48a" (UID: "1e6a9d1a-6a42-44d4-a58c-5177b82bb48a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.943770 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.943630    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-credential-socket" pod "29d290fb-bc3b-4b96-849c-882ab07bf143" (UID: "29d290fb-bc3b-4b96-849c-882ab07bf143"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.944062 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.943856    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-workload-socket" pod "29d290fb-bc3b-4b96-849c-882ab07bf143" (UID: "29d290fb-bc3b-4b96-849c-882ab07bf143"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.944062 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.943916    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-data\") pod \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") "
Nov 01 06:40:00.944062 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.943967    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-token\" (UniqueName: \"kubernetes.io/projected/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-token\") pod \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") "
Nov 01 06:40:00.944759 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.944449    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-workload-socket" pod "1e6a9d1a-6a42-44d4-a58c-5177b82bb48a" (UID: "1e6a9d1a-6a42-44d4-a58c-5177b82bb48a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.944759 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.944514    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istiod-ca-cert\") pod \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") "
Nov 01 06:40:00.944759 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.944565    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29d290fb-bc3b-4b96-849c-882ab07bf143-kube-api-access-wqpzm\" (UniqueName: \"kubernetes.io/projected/29d290fb-bc3b-4b96-849c-882ab07bf143-kube-api-access-wqpzm\") pod \"29d290fb-bc3b-4b96-849c-882ab07bf143\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") "
Nov 01 06:40:00.944759 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.944631    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-kube-api-access-n6vh7\" (UniqueName: \"kubernetes.io/projected/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-kube-api-access-n6vh7\") pod \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\" (UID: \"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a\") "
Nov 01 06:40:00.944759 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.944695    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-workload-certs\") pod \"29d290fb-bc3b-4b96-849c-882ab07bf143\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") "
Nov 01 06:40:00.944759 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.944724    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-podinfo\") pod \"29d290fb-bc3b-4b96-849c-882ab07bf143\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") "
Nov 01 06:40:00.944759 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.944727    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-data" pod "1e6a9d1a-6a42-44d4-a58c-5177b82bb48a" (UID: "1e6a9d1a-6a42-44d4-a58c-5177b82bb48a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.945283 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.945140    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-data" pod "29d290fb-bc3b-4b96-849c-882ab07bf143" (UID: "29d290fb-bc3b-4b96-849c-882ab07bf143"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.945486 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.945463    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istiod-ca-cert" pod "1e6a9d1a-6a42-44d4-a58c-5177b82bb48a" (UID: "1e6a9d1a-6a42-44d4-a58c-5177b82bb48a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:40:00.949513 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.949484    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-podinfo" pod "1e6a9d1a-6a42-44d4-a58c-5177b82bb48a" (UID: "1e6a9d1a-6a42-44d4-a58c-5177b82bb48a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:40:00.949662 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.949639    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-podinfo" pod "29d290fb-bc3b-4b96-849c-882ab07bf143" (UID: "29d290fb-bc3b-4b96-849c-882ab07bf143"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:40:00.949730 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.949694    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29d290fb-bc3b-4b96-849c-882ab07bf143-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29d290fb-bc3b-4b96-849c-882ab07bf143-istiod-ca-cert\") pod \"29d290fb-bc3b-4b96-849c-882ab07bf143\" (UID: \"29d290fb-bc3b-4b96-849c-882ab07bf143\") "
Nov 01 06:40:00.949730 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.949695    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-token" pod "29d290fb-bc3b-4b96-849c-882ab07bf143" (UID: "29d290fb-bc3b-4b96-849c-882ab07bf143"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:00.949864 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.949840    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-token" pod "1e6a9d1a-6a42-44d4-a58c-5177b82bb48a" (UID: "1e6a9d1a-6a42-44d4-a58c-5177b82bb48a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:00.949927 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.949902    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-envoy" pod "1e6a9d1a-6a42-44d4-a58c-5177b82bb48a" (UID: "1e6a9d1a-6a42-44d4-a58c-5177b82bb48a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.950139 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.950115    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-envoy" pod "29d290fb-bc3b-4b96-849c-882ab07bf143" (UID: "29d290fb-bc3b-4b96-849c-882ab07bf143"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.950286 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.950266    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.950399 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.950292    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.950399 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.950309    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.950661 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.950626    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-kube-api-access-n6vh7" pod "1e6a9d1a-6a42-44d4-a58c-5177b82bb48a" (UID: "1e6a9d1a-6a42-44d4-a58c-5177b82bb48a"). InnerVolumeSpecName "kube-api-access-n6vh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:00.950743 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.950723    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29d290fb-bc3b-4b96-849c-882ab07bf143-istiod-ca-cert" pod "29d290fb-bc3b-4b96-849c-882ab07bf143" (UID: "29d290fb-bc3b-4b96-849c-882ab07bf143"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:40:00.950798 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.950779    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-workload-certs" pod "29d290fb-bc3b-4b96-849c-882ab07bf143" (UID: "29d290fb-bc3b-4b96-849c-882ab07bf143"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:40:00.950855 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.950814    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.950855 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.950832    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.950855 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.950846    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.951015 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.950859    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.951015 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.950873    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.951015 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.950886    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.951015 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.950899    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.951015 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.950913    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.951015 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.950927    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqnmm\" (UniqueName: \"kubernetes.io/projected/b92297e9-8458-4787-96f6-e9025d2a6a31-kube-api-access-pqnmm\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.951015 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.950944    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.951015 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.950960    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.951015 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.950974    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92297e9-8458-4787-96f6-e9025d2a6a31-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.951015 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.950987    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.951015 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.951000    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.951015 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.951015    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29d290fb-bc3b-4b96-849c-882ab07bf143-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:00.951657 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:00.951492    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d290fb-bc3b-4b96-849c-882ab07bf143-kube-api-access-wqpzm" pod "29d290fb-bc3b-4b96-849c-882ab07bf143" (UID: "29d290fb-bc3b-4b96-849c-882ab07bf143"). InnerVolumeSpecName "kube-api-access-wqpzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:40:01.052463 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.052417    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqpzm\" (UniqueName: \"kubernetes.io/projected/29d290fb-bc3b-4b96-849c-882ab07bf143-kube-api-access-wqpzm\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:01.052463 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.052456    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6vh7\" (UniqueName: \"kubernetes.io/projected/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a-kube-api-access-n6vh7\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:01.052874 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.052471    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29d290fb-bc3b-4b96-849c-882ab07bf143-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:01.052874 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.052485    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29d290fb-bc3b-4b96-849c-882ab07bf143-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:40:01.285550 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.285469    5619 generic.go:356] "Generic (PLEG): container finished" podID="b92297e9-8458-4787-96f6-e9025d2a6a31" containerID="28f4379fbd0c224c4de13336cdc30f2aabf545e20243d1deddd8b775c277f852" exitCode=0
Nov 01 06:40:01.285550 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.285519    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"
Nov 01 06:40:01.285776 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.285551    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94" event={"ID":"b92297e9-8458-4787-96f6-e9025d2a6a31","Type":"ContainerDied","Data":"28f4379fbd0c224c4de13336cdc30f2aabf545e20243d1deddd8b775c277f852"}
Nov 01 06:40:01.285776 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.285605    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94" event={"ID":"b92297e9-8458-4787-96f6-e9025d2a6a31","Type":"ContainerDied","Data":"9286f22540c5308189585032fc2c346e30093275ec76a02f65ec4d67ae67a378"}
Nov 01 06:40:01.285776 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.285630    5619 scope.go:122] "RemoveContainer" containerID="28f4379fbd0c224c4de13336cdc30f2aabf545e20243d1deddd8b775c277f852"
Nov 01 06:40:01.287082 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.287062    5619 generic.go:356] "Generic (PLEG): container finished" podID="29d290fb-bc3b-4b96-849c-882ab07bf143" containerID="873a3cbebf6832b63c7f158a602e0bcd85ea76e8bed1ee9dc707f0700a364191" exitCode=0
Nov 01 06:40:01.287176 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.287158    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7" event={"ID":"29d290fb-bc3b-4b96-849c-882ab07bf143","Type":"ContainerDied","Data":"873a3cbebf6832b63c7f158a602e0bcd85ea76e8bed1ee9dc707f0700a364191"}
Nov 01 06:40:01.287240 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.287178    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"
Nov 01 06:40:01.287240 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.287202    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7" event={"ID":"29d290fb-bc3b-4b96-849c-882ab07bf143","Type":"ContainerDied","Data":"5d95899bd972159c8bf0f5159fed817c3a853b7c17c85969c9f62f9a63800558"}
Nov 01 06:40:01.289216 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.289192    5619 generic.go:356] "Generic (PLEG): container finished" podID="1e6a9d1a-6a42-44d4-a58c-5177b82bb48a" containerID="92ef82ae79f04d55d350d484d0b2bac03daaac769633945083901b8213f8fb07" exitCode=0
Nov 01 06:40:01.289893 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.289255    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"
Nov 01 06:40:01.289893 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.289266    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk" event={"ID":"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a","Type":"ContainerDied","Data":"92ef82ae79f04d55d350d484d0b2bac03daaac769633945083901b8213f8fb07"}
Nov 01 06:40:01.289893 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.289304    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk" event={"ID":"1e6a9d1a-6a42-44d4-a58c-5177b82bb48a","Type":"ContainerDied","Data":"2ee5e265c922401fb1233d1fd5f8ea43ef81573de3ac2afb25255f36ee71ad85"}
Nov 01 06:40:01.296676 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.296659    5619 scope.go:122] "RemoveContainer" containerID="28f4379fbd0c224c4de13336cdc30f2aabf545e20243d1deddd8b775c277f852"
Nov 01 06:40:01.296948 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:40:01.296923    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f4379fbd0c224c4de13336cdc30f2aabf545e20243d1deddd8b775c277f852\": container with ID starting with 28f4379fbd0c224c4de13336cdc30f2aabf545e20243d1deddd8b775c277f852 not found: ID does not exist" containerID="28f4379fbd0c224c4de13336cdc30f2aabf545e20243d1deddd8b775c277f852"
Nov 01 06:40:01.297045 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.296957    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f4379fbd0c224c4de13336cdc30f2aabf545e20243d1deddd8b775c277f852"} err="failed to get container status \"28f4379fbd0c224c4de13336cdc30f2aabf545e20243d1deddd8b775c277f852\": rpc error: code = NotFound desc = could not find container \"28f4379fbd0c224c4de13336cdc30f2aabf545e20243d1deddd8b775c277f852\": container with ID starting with 28f4379fbd0c224c4de13336cdc30f2aabf545e20243d1deddd8b775c277f852 not found: ID does not exist"
Nov 01 06:40:01.297045 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.296975    5619 scope.go:122] "RemoveContainer" containerID="873a3cbebf6832b63c7f158a602e0bcd85ea76e8bed1ee9dc707f0700a364191"
Nov 01 06:40:01.305411 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.305394    5619 scope.go:122] "RemoveContainer" containerID="873a3cbebf6832b63c7f158a602e0bcd85ea76e8bed1ee9dc707f0700a364191"
Nov 01 06:40:01.305665 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:40:01.305648    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"873a3cbebf6832b63c7f158a602e0bcd85ea76e8bed1ee9dc707f0700a364191\": container with ID starting with 873a3cbebf6832b63c7f158a602e0bcd85ea76e8bed1ee9dc707f0700a364191 not found: ID does not exist" containerID="873a3cbebf6832b63c7f158a602e0bcd85ea76e8bed1ee9dc707f0700a364191"
Nov 01 06:40:01.305726 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.305672    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873a3cbebf6832b63c7f158a602e0bcd85ea76e8bed1ee9dc707f0700a364191"} err="failed to get container status \"873a3cbebf6832b63c7f158a602e0bcd85ea76e8bed1ee9dc707f0700a364191\": rpc error: code = NotFound desc = could not find container \"873a3cbebf6832b63c7f158a602e0bcd85ea76e8bed1ee9dc707f0700a364191\": container with ID starting with 873a3cbebf6832b63c7f158a602e0bcd85ea76e8bed1ee9dc707f0700a364191 not found: ID does not exist"
Nov 01 06:40:01.305726 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.305688    5619 scope.go:122] "RemoveContainer" containerID="92ef82ae79f04d55d350d484d0b2bac03daaac769633945083901b8213f8fb07"
Nov 01 06:40:01.313851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.313834    5619 scope.go:122] "RemoveContainer" containerID="92ef82ae79f04d55d350d484d0b2bac03daaac769633945083901b8213f8fb07"
Nov 01 06:40:01.314099 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:40:01.314081    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92ef82ae79f04d55d350d484d0b2bac03daaac769633945083901b8213f8fb07\": container with ID starting with 92ef82ae79f04d55d350d484d0b2bac03daaac769633945083901b8213f8fb07 not found: ID does not exist" containerID="92ef82ae79f04d55d350d484d0b2bac03daaac769633945083901b8213f8fb07"
Nov 01 06:40:01.314162 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.314106    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92ef82ae79f04d55d350d484d0b2bac03daaac769633945083901b8213f8fb07"} err="failed to get container status \"92ef82ae79f04d55d350d484d0b2bac03daaac769633945083901b8213f8fb07\": rpc error: code = NotFound desc = could not find container \"92ef82ae79f04d55d350d484d0b2bac03daaac769633945083901b8213f8fb07\": container with ID starting with 92ef82ae79f04d55d350d484d0b2bac03daaac769633945083901b8213f8fb07 not found: ID does not exist"
Nov 01 06:40:01.459015 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.458985    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"]
Nov 01 06:40:01.465413 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.465390    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-c75dk"]
Nov 01 06:40:01.576738 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.576663    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"]
Nov 01 06:40:01.582944 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.582919    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-vvd94"]
Nov 01 06:40:01.694367 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.694337    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"]
Nov 01 06:40:01.699165 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:01.699135    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c0662f25-openshift-default-86d794b649-8j9x7"]
Nov 01 06:40:02.972968 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:02.972935    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e6a9d1a-6a42-44d4-a58c-5177b82bb48a" path="/var/lib/kubelet/pods/1e6a9d1a-6a42-44d4-a58c-5177b82bb48a/volumes"
Nov 01 06:40:02.973418 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:02.973402    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29d290fb-bc3b-4b96-849c-882ab07bf143" path="/var/lib/kubelet/pods/29d290fb-bc3b-4b96-849c-882ab07bf143/volumes"
Nov 01 06:40:02.973841 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:02.973828    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b92297e9-8458-4787-96f6-e9025d2a6a31" path="/var/lib/kubelet/pods/b92297e9-8458-4787-96f6-e9025d2a6a31/volumes"
Nov 01 06:40:04.693090 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:04.693042    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:09.461415 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:09.461357    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:13.017922 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:13.017833    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:13.164852 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:13.164803    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:13.316522 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:13.316435    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:14.250148 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:14.250100    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:14.460512 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:14.460466    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:14.616477 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:14.616439    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:19.060344 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:19.060289    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:19.945570 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:19.945530    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:24.073813 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:24.073769    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:25.355836 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:25.355794    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:25.554172 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:25.554124    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:25.881625 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:40:25.881587    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:40:28.880812 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:28.880763    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:30.434961 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:30.434921    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:32.129285 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:32.129253    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:33.273862 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:33.273816    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:33.771275 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:33.771247    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:34.964677 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:34.964636    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:38.261750 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:38.261715    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:38.668035 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:38.667995    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:43.458128 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:43.458072    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:48.194986 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:48.194946    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:52.954502 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:52.954453    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:40:55.880043 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:40:55.880007    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:40:57.654820 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:40:57.654781    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:02.567194 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:02.567160    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:05.280211 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:05.280165    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:05.732161 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:05.732123    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:07.258654 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:07.258613    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:10.382435 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:10.382388    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:10.784008 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:10.783918    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:11.954848 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:11.954808    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:16.685624 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:16.685590    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:21.431581 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:21.431543    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:25.879511 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:41:25.879466    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:41:26.086389 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:26.086345    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:30.790437 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:30.790391    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:30.935217 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:30.935176    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:35.307731 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:35.307694    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:35.657238 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:35.657201    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:40.377476 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:40.377437    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:45.151005 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:45.150967    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:47.732670 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:47.732631    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:48.379879 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:48.379841    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g"]
Nov 01 06:41:48.631605 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:48.631516    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:48.772567 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:48.772519    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:50.221421 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:50.221387    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:53.380663 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:53.380607    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g" podUID="c1e533b8-8047-477b-a358-c72cb0b096a1" containerName="payload-pre-processing" containerID="cri-o://f9c54eb9d92b40fd962fd9392006d24bbee4104bf81df486d438d15612901514" gracePeriod=25
Nov 01 06:41:53.465711 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:53.465683    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:53.662185 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:53.662155    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g"
Nov 01 06:41:53.664394 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:53.664372    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e533b8-8047-477b-a358-c72cb0b096a1-kube-api-access-wrdrk\" (UniqueName: \"kubernetes.io/projected/c1e533b8-8047-477b-a358-c72cb0b096a1-kube-api-access-wrdrk\") pod \"c1e533b8-8047-477b-a358-c72cb0b096a1\" (UID: \"c1e533b8-8047-477b-a358-c72cb0b096a1\") "
Nov 01 06:41:53.664543 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:53.664523    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1e533b8-8047-477b-a358-c72cb0b096a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e533b8-8047-477b-a358-c72cb0b096a1-plugins-config-volume\") pod \"c1e533b8-8047-477b-a358-c72cb0b096a1\" (UID: \"c1e533b8-8047-477b-a358-c72cb0b096a1\") "
Nov 01 06:41:53.664927 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:53.664903    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e533b8-8047-477b-a358-c72cb0b096a1-plugins-config-volume" pod "c1e533b8-8047-477b-a358-c72cb0b096a1" (UID: "c1e533b8-8047-477b-a358-c72cb0b096a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:41:53.670720 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:53.670691    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e533b8-8047-477b-a358-c72cb0b096a1-kube-api-access-wrdrk" pod "c1e533b8-8047-477b-a358-c72cb0b096a1" (UID: "c1e533b8-8047-477b-a358-c72cb0b096a1"). InnerVolumeSpecName "kube-api-access-wrdrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:41:53.765678 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:53.765646    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e533b8-8047-477b-a358-c72cb0b096a1-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:41:53.765678 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:53.765671    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrdrk\" (UniqueName: \"kubernetes.io/projected/c1e533b8-8047-477b-a358-c72cb0b096a1-kube-api-access-wrdrk\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:41:54.058717 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:54.058632    5619 generic.go:356] "Generic (PLEG): container finished" podID="c1e533b8-8047-477b-a358-c72cb0b096a1" containerID="f9c54eb9d92b40fd962fd9392006d24bbee4104bf81df486d438d15612901514" exitCode=0
Nov 01 06:41:54.058717 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:54.058681    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g"
Nov 01 06:41:54.058904 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:54.058718    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g" event={"ID":"c1e533b8-8047-477b-a358-c72cb0b096a1","Type":"ContainerDied","Data":"f9c54eb9d92b40fd962fd9392006d24bbee4104bf81df486d438d15612901514"}
Nov 01 06:41:54.058904 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:54.058777    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g" event={"ID":"c1e533b8-8047-477b-a358-c72cb0b096a1","Type":"ContainerDied","Data":"39f72cf319bed752264f8cbd14c1d9c3560f86c276ddf8288f30c16b59bead9c"}
Nov 01 06:41:54.058904 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:54.058802    5619 scope.go:122] "RemoveContainer" containerID="f9c54eb9d92b40fd962fd9392006d24bbee4104bf81df486d438d15612901514"
Nov 01 06:41:54.068720 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:54.068702    5619 scope.go:122] "RemoveContainer" containerID="f9c54eb9d92b40fd962fd9392006d24bbee4104bf81df486d438d15612901514"
Nov 01 06:41:54.069044 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:41:54.069024    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9c54eb9d92b40fd962fd9392006d24bbee4104bf81df486d438d15612901514\": container with ID starting with f9c54eb9d92b40fd962fd9392006d24bbee4104bf81df486d438d15612901514 not found: ID does not exist" containerID="f9c54eb9d92b40fd962fd9392006d24bbee4104bf81df486d438d15612901514"
Nov 01 06:41:54.069101 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:54.069050    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c54eb9d92b40fd962fd9392006d24bbee4104bf81df486d438d15612901514"} err="failed to get container status \"f9c54eb9d92b40fd962fd9392006d24bbee4104bf81df486d438d15612901514\": rpc error: code = NotFound desc = could not find container \"f9c54eb9d92b40fd962fd9392006d24bbee4104bf81df486d438d15612901514\": container with ID starting with f9c54eb9d92b40fd962fd9392006d24bbee4104bf81df486d438d15612901514 not found: ID does not exist"
Nov 01 06:41:54.225274 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:54.225242    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g"]
Nov 01 06:41:54.231208 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:54.231180    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c673e5-54cd8ff65f-h6c8g"]
Nov 01 06:41:54.962210 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:54.962178    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1e533b8-8047-477b-a358-c72cb0b096a1" path="/var/lib/kubelet/pods/c1e533b8-8047-477b-a358-c72cb0b096a1/volumes"
Nov 01 06:41:55.070360 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:55.070313    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:41:55.879233 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:41:55.879196    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:41:59.771172 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:41:59.771134    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:04.575494 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:04.575454    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:09.354426 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:09.354391    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:13.741064 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:13.741028    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:14.072475 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:14.072396    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:18.300974 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:18.300935    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:19.586092 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:19.586029    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:20.052526 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:20.052488    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:24.746148 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:24.746110    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:25.542174 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:25.542144    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:25.682601 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:25.682561    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:25.882048 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:42:25.882004    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:42:27.139642 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:27.139609    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:27.795482 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:27.795432    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:27.797074 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:27.797045    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"]
Nov 01 06:42:27.797269 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:27.797248    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6" podUID="9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" containerName="istio-proxy" containerID="cri-o://349417e0f0d0fc1c8a41c5bd39281efc96e5db156544cf2df62191d451fa514a" gracePeriod=30
Nov 01 06:42:28.410149 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:28.410117    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:29.040595 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:29.040559    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:29.310398 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:29.310295    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:29.768734 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:29.768692    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:30.397855 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:30.397821    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:30.544808 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:30.544757    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:30.689417 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:30.689303    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:31.233045 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:31.233007    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:33.062204 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.062139    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:42:33.151909 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.151876    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-podinfo\") pod \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") "
Nov 01 06:42:33.152100 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.151931    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-token\" (UniqueName: \"kubernetes.io/projected/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-token\") pod \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") "
Nov 01 06:42:33.152100 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.151970    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-credential-socket\") pod \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") "
Nov 01 06:42:33.152100 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.152013    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-workload-socket\") pod \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") "
Nov 01 06:42:33.152100 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.152033    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-kube-api-access-6xpcw\" (UniqueName: \"kubernetes.io/projected/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-kube-api-access-6xpcw\") pod \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") "
Nov 01 06:42:33.152100 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.152095    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-envoy\") pod \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") "
Nov 01 06:42:33.152421 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.152140    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-workload-certs\") pod \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") "
Nov 01 06:42:33.152421 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.152285    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-data\") pod \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") "
Nov 01 06:42:33.152421 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.152352    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-workload-socket" pod "9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" (UID: "9ff3bbf5-47d7-4a26-95c6-ed22acc794c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:42:33.152421 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.152365    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istiod-ca-cert\") pod \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\" (UID: \"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3\") "
Nov 01 06:42:33.152785 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.152367    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-credential-socket" pod "9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" (UID: "9ff3bbf5-47d7-4a26-95c6-ed22acc794c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:42:33.152785 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.152685    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-workload-certs" pod "9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" (UID: "9ff3bbf5-47d7-4a26-95c6-ed22acc794c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:42:33.152938 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.152906    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istiod-ca-cert" pod "9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" (UID: "9ff3bbf5-47d7-4a26-95c6-ed22acc794c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:42:33.152938 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.152915    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:33.153057 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.152940    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:33.153057 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.152951    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:33.153169 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.153071    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-data" pod "9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" (UID: "9ff3bbf5-47d7-4a26-95c6-ed22acc794c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:42:33.154333 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.154302    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-token" pod "9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" (UID: "9ff3bbf5-47d7-4a26-95c6-ed22acc794c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:42:33.154457 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.154441    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-envoy" pod "9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" (UID: "9ff3bbf5-47d7-4a26-95c6-ed22acc794c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:42:33.154633 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.154609    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-podinfo" pod "9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" (UID: "9ff3bbf5-47d7-4a26-95c6-ed22acc794c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:42:33.154737 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.154637    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-kube-api-access-6xpcw" pod "9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" (UID: "9ff3bbf5-47d7-4a26-95c6-ed22acc794c3"). InnerVolumeSpecName "kube-api-access-6xpcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:42:33.253866 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.253830    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xpcw\" (UniqueName: \"kubernetes.io/projected/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-kube-api-access-6xpcw\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:33.253866 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.253861    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:33.253866 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.253874    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:33.254188 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.253883    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:33.254188 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.253894    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:33.254188 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.253903    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:42:33.339883 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.339782    5619 generic.go:356] "Generic (PLEG): container finished" podID="9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" containerID="349417e0f0d0fc1c8a41c5bd39281efc96e5db156544cf2df62191d451fa514a" exitCode=0
Nov 01 06:42:33.339883 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.339833    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"
Nov 01 06:42:33.339883 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.339853    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6" event={"ID":"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3","Type":"ContainerDied","Data":"349417e0f0d0fc1c8a41c5bd39281efc96e5db156544cf2df62191d451fa514a"}
Nov 01 06:42:33.340160 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.339906    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6" event={"ID":"9ff3bbf5-47d7-4a26-95c6-ed22acc794c3","Type":"ContainerDied","Data":"14a5f5cf295177838881db81eaf77958fb0c2db24b0b8a051719c8b9ffaa3811"}
Nov 01 06:42:33.340160 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.339935    5619 scope.go:122] "RemoveContainer" containerID="349417e0f0d0fc1c8a41c5bd39281efc96e5db156544cf2df62191d451fa514a"
Nov 01 06:42:33.351094 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.351071    5619 scope.go:122] "RemoveContainer" containerID="ff1aca6046adac53531db1e0751b16de9bba8f3bffcb5895fa79c9b096cf703c"
Nov 01 06:42:33.360438 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.360416    5619 scope.go:122] "RemoveContainer" containerID="349417e0f0d0fc1c8a41c5bd39281efc96e5db156544cf2df62191d451fa514a"
Nov 01 06:42:33.360725 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:42:33.360692    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"349417e0f0d0fc1c8a41c5bd39281efc96e5db156544cf2df62191d451fa514a\": container with ID starting with 349417e0f0d0fc1c8a41c5bd39281efc96e5db156544cf2df62191d451fa514a not found: ID does not exist" containerID="349417e0f0d0fc1c8a41c5bd39281efc96e5db156544cf2df62191d451fa514a"
Nov 01 06:42:33.360857 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.360728    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"349417e0f0d0fc1c8a41c5bd39281efc96e5db156544cf2df62191d451fa514a"} err="failed to get container status \"349417e0f0d0fc1c8a41c5bd39281efc96e5db156544cf2df62191d451fa514a\": rpc error: code = NotFound desc = could not find container \"349417e0f0d0fc1c8a41c5bd39281efc96e5db156544cf2df62191d451fa514a\": container with ID starting with 349417e0f0d0fc1c8a41c5bd39281efc96e5db156544cf2df62191d451fa514a not found: ID does not exist"
Nov 01 06:42:33.360857 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.360753    5619 scope.go:122] "RemoveContainer" containerID="ff1aca6046adac53531db1e0751b16de9bba8f3bffcb5895fa79c9b096cf703c"
Nov 01 06:42:33.361037 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:42:33.361017    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff1aca6046adac53531db1e0751b16de9bba8f3bffcb5895fa79c9b096cf703c\": container with ID starting with ff1aca6046adac53531db1e0751b16de9bba8f3bffcb5895fa79c9b096cf703c not found: ID does not exist" containerID="ff1aca6046adac53531db1e0751b16de9bba8f3bffcb5895fa79c9b096cf703c"
Nov 01 06:42:33.361079 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.361045    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1aca6046adac53531db1e0751b16de9bba8f3bffcb5895fa79c9b096cf703c"} err="failed to get container status \"ff1aca6046adac53531db1e0751b16de9bba8f3bffcb5895fa79c9b096cf703c\": rpc error: code = NotFound desc = could not find container \"ff1aca6046adac53531db1e0751b16de9bba8f3bffcb5895fa79c9b096cf703c\": container with ID starting with ff1aca6046adac53531db1e0751b16de9bba8f3bffcb5895fa79c9b096cf703c not found: ID does not exist"
Nov 01 06:42:33.513117 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.513080    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"]
Nov 01 06:42:33.516929 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:33.516897    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-c673e5-openshift-default-d97d76cb9-cz6v6"]
Nov 01 06:42:34.351470 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:34.351414    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:34.610936 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:34.610839    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:34.962958 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:34.962879    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" path="/var/lib/kubelet/pods/9ff3bbf5-47d7-4a26-95c6-ed22acc794c3/volumes"
Nov 01 06:42:35.662543 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:35.662503    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:39.320984 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:39.320926    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:44.165662 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:44.165622    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:48.968521 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:48.968478    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:53.762198 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:53.762156    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:42:55.881105 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:42:55.881062    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:42:58.465627 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:42:58.465581    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:03.177659 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:03.177614    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:04.288518 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:04.288428    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:04.464345 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:04.464238    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:04.846118 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:04.846077    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:04.985692 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:04.985661    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:07.963177 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:07.963134    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:09.949300 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:09.949264    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:12.776739 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:12.776634    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:17.471453 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:17.471414    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:20.785377 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:20.785342    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"]
Nov 01 06:43:20.878337 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:20.878279    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:22.201820 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:22.201784    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:25.879901 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:43:25.879861    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:43:27.054830 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:27.054796    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:27.201220 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:27.201190    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:28.252295 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.252258    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"]
Nov 01 06:43:28.253724 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.253696    5619 cpu_manager.go:414] "containerMap: removing container" podUID="304878ad-a41a-4349-8ec7-3c7d38f4efbc" containerName="maas-api"
Nov 01 06:43:28.253870 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.253858    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="304878ad-a41a-4349-8ec7-3c7d38f4efbc" containerName="maas-api"
Nov 01 06:43:28.255183 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.255160    5619 cpu_manager.go:414] "containerMap: removing container" podUID="304878ad-a41a-4349-8ec7-3c7d38f4efbc" containerName="maas-api"
Nov 01 06:43:28.255317 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.255305    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="304878ad-a41a-4349-8ec7-3c7d38f4efbc" containerName="maas-api"
Nov 01 06:43:28.255449 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.255437    5619 cpu_manager.go:414] "containerMap: removing container" podUID="9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" containerName="istio-proxy"
Nov 01 06:43:28.255519 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.255509    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" containerName="istio-proxy"
Nov 01 06:43:28.255599 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.255590    5619 cpu_manager.go:414] "containerMap: removing container" podUID="1e6a9d1a-6a42-44d4-a58c-5177b82bb48a" containerName="istio-proxy"
Nov 01 06:43:28.255669 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.255659    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e6a9d1a-6a42-44d4-a58c-5177b82bb48a" containerName="istio-proxy"
Nov 01 06:43:28.255740 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.255732    5619 cpu_manager.go:414] "containerMap: removing container" podUID="b92297e9-8458-4787-96f6-e9025d2a6a31" containerName="istio-proxy"
Nov 01 06:43:28.255808 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.255799    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b92297e9-8458-4787-96f6-e9025d2a6a31" containerName="istio-proxy"
Nov 01 06:43:28.255880 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.255871    5619 cpu_manager.go:414] "containerMap: removing container" podUID="29d290fb-bc3b-4b96-849c-882ab07bf143" containerName="istio-proxy"
Nov 01 06:43:28.255948 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.255937    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d290fb-bc3b-4b96-849c-882ab07bf143" containerName="istio-proxy"
Nov 01 06:43:28.258030 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.258012    5619 cpu_manager.go:414] "containerMap: removing container" podUID="c1e533b8-8047-477b-a358-c72cb0b096a1" containerName="payload-pre-processing"
Nov 01 06:43:28.258137 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.258123    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e533b8-8047-477b-a358-c72cb0b096a1" containerName="payload-pre-processing"
Nov 01 06:43:28.258396 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.258381    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="304878ad-a41a-4349-8ec7-3c7d38f4efbc" containerName="maas-api"
Nov 01 06:43:28.258510 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.258499    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1e533b8-8047-477b-a358-c72cb0b096a1" containerName="payload-pre-processing"
Nov 01 06:43:28.258594 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.258584    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" containerName="istio-proxy"
Nov 01 06:43:28.258676 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.258667    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="304878ad-a41a-4349-8ec7-3c7d38f4efbc" containerName="maas-api"
Nov 01 06:43:28.258753 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.258742    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="304878ad-a41a-4349-8ec7-3c7d38f4efbc" containerName="maas-api"
Nov 01 06:43:28.258835 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.258825    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e6a9d1a-6a42-44d4-a58c-5177b82bb48a" containerName="istio-proxy"
Nov 01 06:43:28.258911 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.258901    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="b92297e9-8458-4787-96f6-e9025d2a6a31" containerName="istio-proxy"
Nov 01 06:43:28.258989 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.258980    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d290fb-bc3b-4b96-849c-882ab07bf143" containerName="istio-proxy"
Nov 01 06:43:28.274370 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.274339    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"
Nov 01 06:43:28.308952 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.308904    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm9lh\" (UniqueName: \"kubernetes.io/projected/623f7715-a2cc-41d3-813e-236c25f7925d-kube-api-access-vm9lh\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn\" (UID: \"623f7715-a2cc-41d3-813e-236c25f7925d\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"
Nov 01 06:43:28.309143 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.308979    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/623f7715-a2cc-41d3-813e-236c25f7925d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn\" (UID: \"623f7715-a2cc-41d3-813e-236c25f7925d\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"
Nov 01 06:43:28.309143 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.309124    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/623f7715-a2cc-41d3-813e-236c25f7925d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn\" (UID: \"623f7715-a2cc-41d3-813e-236c25f7925d\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"
Nov 01 06:43:28.309263 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.309157    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/623f7715-a2cc-41d3-813e-236c25f7925d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn\" (UID: \"623f7715-a2cc-41d3-813e-236c25f7925d\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"
Nov 01 06:43:28.319764 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.319733    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"]
Nov 01 06:43:28.410936 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.410899    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm9lh\" (UniqueName: \"kubernetes.io/projected/623f7715-a2cc-41d3-813e-236c25f7925d-kube-api-access-vm9lh\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn\" (UID: \"623f7715-a2cc-41d3-813e-236c25f7925d\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"
Nov 01 06:43:28.411153 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.410945    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/623f7715-a2cc-41d3-813e-236c25f7925d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn\" (UID: \"623f7715-a2cc-41d3-813e-236c25f7925d\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"
Nov 01 06:43:28.411153 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.410997    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/623f7715-a2cc-41d3-813e-236c25f7925d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn\" (UID: \"623f7715-a2cc-41d3-813e-236c25f7925d\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"
Nov 01 06:43:28.411153 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.411021    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/623f7715-a2cc-41d3-813e-236c25f7925d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn\" (UID: \"623f7715-a2cc-41d3-813e-236c25f7925d\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"
Nov 01 06:43:28.411954 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.411920    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/623f7715-a2cc-41d3-813e-236c25f7925d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn\" (UID: \"623f7715-a2cc-41d3-813e-236c25f7925d\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"
Nov 01 06:43:28.412108 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.412088    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/623f7715-a2cc-41d3-813e-236c25f7925d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn\" (UID: \"623f7715-a2cc-41d3-813e-236c25f7925d\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"
Nov 01 06:43:28.414289 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.414263    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/623f7715-a2cc-41d3-813e-236c25f7925d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn\" (UID: \"623f7715-a2cc-41d3-813e-236c25f7925d\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"
Nov 01 06:43:28.472716 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.472685    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm9lh\" (UniqueName: \"kubernetes.io/projected/623f7715-a2cc-41d3-813e-236c25f7925d-kube-api-access-vm9lh\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn\" (UID: \"623f7715-a2cc-41d3-813e-236c25f7925d\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"
Nov 01 06:43:28.592583 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.592556    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"
Nov 01 06:43:28.970160 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:28.970127    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"]
Nov 01 06:43:28.971235 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:43:28.971210    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623f7715_a2cc_41d3_813e_236c25f7925d.slice/crio-19cf23ae462e973f934323584ed1468406fb62a9ae306a6d937f7e65efb11101 WatchSource:0}: Error finding container 19cf23ae462e973f934323584ed1468406fb62a9ae306a6d937f7e65efb11101: Status 404 returned error can't find the container with id 19cf23ae462e973f934323584ed1468406fb62a9ae306a6d937f7e65efb11101
Nov 01 06:43:29.737740 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:29.737708    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn" event={"ID":"623f7715-a2cc-41d3-813e-236c25f7925d","Type":"ContainerStarted","Data":"79046f108a85eb1efeb60ba1c41cab8ca745f4516bf7577b5929660f586f05cd"}
Nov 01 06:43:29.738166 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:29.737763    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn" event={"ID":"623f7715-a2cc-41d3-813e-236c25f7925d","Type":"ContainerStarted","Data":"19cf23ae462e973f934323584ed1468406fb62a9ae306a6d937f7e65efb11101"}
Nov 01 06:43:29.738166 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:29.737791    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"
Nov 01 06:43:29.857711 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:29.857642    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn" podStartSLOduration=1.8576197159999999 podStartE2EDuration="1.857619716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:43:29.855299373 +0000 UTC m=+3447.066122883" watchObservedRunningTime="2026-11-01 06:43:29.857619716 +0000 UTC m=+3447.068443238"
Nov 01 06:43:31.598087 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:31.598047    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:31.789419 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:31.789376    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:35.344936 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:35.344899    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:35.805589 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:35.805494    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j" podUID="81cdf4f9-7e3c-47c0-b669-e06ec353fdf6" containerName="main" containerID="cri-o://64b602c1bd3dfc3eafb5a2b92d2a57b2039ffe1e5242d07f85e46f6065047b68" gracePeriod=45
Nov 01 06:43:36.009924 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.009854    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:36.066034 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.065971    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:43:36.210879 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.210847    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-kube-api-access-spkjd\" (UniqueName: \"kubernetes.io/projected/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-kube-api-access-spkjd\") pod \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") "
Nov 01 06:43:36.211051 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.210926    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-model-cache\") pod \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") "
Nov 01 06:43:36.211051 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.210992    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-tmp-dir\") pod \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") "
Nov 01 06:43:36.211189 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.211054    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-dshm\") pod \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") "
Nov 01 06:43:36.211189 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.211162    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-tls-certs\" (UniqueName: \"kubernetes.io/secret/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-tls-certs\") pod \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") "
Nov 01 06:43:36.211302 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.211224    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-home\" (UniqueName: \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-home\") pod \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\" (UID: \"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6\") "
Nov 01 06:43:36.211302 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.211248    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-tmp-dir" pod "81cdf4f9-7e3c-47c0-b669-e06ec353fdf6" (UID: "81cdf4f9-7e3c-47c0-b669-e06ec353fdf6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:43:36.211302 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.211263    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-model-cache" pod "81cdf4f9-7e3c-47c0-b669-e06ec353fdf6" (UID: "81cdf4f9-7e3c-47c0-b669-e06ec353fdf6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:43:36.211546 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.211528    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-home" pod "81cdf4f9-7e3c-47c0-b669-e06ec353fdf6" (UID: "81cdf4f9-7e3c-47c0-b669-e06ec353fdf6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:43:36.211714 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.211690    5619 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-home\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:43:36.211714 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.211711    5619 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-model-cache\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:43:36.211883 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.211724    5619 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-tmp-dir\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:43:36.213628 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.213601    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-dshm" pod "81cdf4f9-7e3c-47c0-b669-e06ec353fdf6" (UID: "81cdf4f9-7e3c-47c0-b669-e06ec353fdf6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:43:36.213628 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.213612    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-tls-certs" pod "81cdf4f9-7e3c-47c0-b669-e06ec353fdf6" (UID: "81cdf4f9-7e3c-47c0-b669-e06ec353fdf6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:43:36.213783 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.213655    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-kube-api-access-spkjd" pod "81cdf4f9-7e3c-47c0-b669-e06ec353fdf6" (UID: "81cdf4f9-7e3c-47c0-b669-e06ec353fdf6"). InnerVolumeSpecName "kube-api-access-spkjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:43:36.312609 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.312576    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spkjd\" (UniqueName: \"kubernetes.io/projected/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-kube-api-access-spkjd\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:43:36.312609 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.312604    5619 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-dshm\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:43:36.312609 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.312614    5619 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6-tls-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:43:36.510492 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.510449    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:36.783056 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.782972    5619 generic.go:356] "Generic (PLEG): container finished" podID="81cdf4f9-7e3c-47c0-b669-e06ec353fdf6" containerID="64b602c1bd3dfc3eafb5a2b92d2a57b2039ffe1e5242d07f85e46f6065047b68" exitCode=0
Nov 01 06:43:36.783056 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.783021    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"
Nov 01 06:43:36.783264 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.783032    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j" event={"ID":"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6","Type":"ContainerDied","Data":"64b602c1bd3dfc3eafb5a2b92d2a57b2039ffe1e5242d07f85e46f6065047b68"}
Nov 01 06:43:36.783264 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.783124    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j" event={"ID":"81cdf4f9-7e3c-47c0-b669-e06ec353fdf6","Type":"ContainerDied","Data":"7ba62fbc669343de11d4aacd3e5b740aecdcd3ddfe7a651ebfd521fae9727bc4"}
Nov 01 06:43:36.783264 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.783154    5619 scope.go:122] "RemoveContainer" containerID="64b602c1bd3dfc3eafb5a2b92d2a57b2039ffe1e5242d07f85e46f6065047b68"
Nov 01 06:43:36.793740 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.793719    5619 scope.go:122] "RemoveContainer" containerID="64b602c1bd3dfc3eafb5a2b92d2a57b2039ffe1e5242d07f85e46f6065047b68"
Nov 01 06:43:36.794064 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:43:36.794043    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b602c1bd3dfc3eafb5a2b92d2a57b2039ffe1e5242d07f85e46f6065047b68\": container with ID starting with 64b602c1bd3dfc3eafb5a2b92d2a57b2039ffe1e5242d07f85e46f6065047b68 not found: ID does not exist" containerID="64b602c1bd3dfc3eafb5a2b92d2a57b2039ffe1e5242d07f85e46f6065047b68"
Nov 01 06:43:36.794122 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.794074    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b602c1bd3dfc3eafb5a2b92d2a57b2039ffe1e5242d07f85e46f6065047b68"} err="failed to get container status \"64b602c1bd3dfc3eafb5a2b92d2a57b2039ffe1e5242d07f85e46f6065047b68\": rpc error: code = NotFound desc = could not find container \"64b602c1bd3dfc3eafb5a2b92d2a57b2039ffe1e5242d07f85e46f6065047b68\": container with ID starting with 64b602c1bd3dfc3eafb5a2b92d2a57b2039ffe1e5242d07f85e46f6065047b68 not found: ID does not exist"
Nov 01 06:43:36.950610 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.950576    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"]
Nov 01 06:43:36.962203 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:36.962177    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-d7d444/rate-test-model-d7d444-kserve-79795bcd69-mbn6j"]
Nov 01 06:43:38.962522 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:38.962492    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81cdf4f9-7e3c-47c0-b669-e06ec353fdf6" path="/var/lib/kubelet/pods/81cdf4f9-7e3c-47c0-b669-e06ec353fdf6/volumes"
Nov 01 06:43:40.747056 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:40.747019    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn" podUID="623f7715-a2cc-41d3-813e-236c25f7925d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:43:41.266776 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:41.266745    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:44.279519 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:44.279485    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:44.476795 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:44.476760    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:46.072875 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:46.072837    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:47.247401 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:47.247368    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:48.704090 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:48.704042    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:49.299090 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:49.299052    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:49.599227 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:49.599185    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:50.745666 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:50.745631    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn" podUID="623f7715-a2cc-41d3-813e-236c25f7925d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:43:50.898115 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:50.898075    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:53.574154 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:53.574111    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:53.744446 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:53.744406    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:53.947394 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:53.947339    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:55.255108 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:55.255063    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:55.529570 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:55.529486    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:55.879253 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:43:55.879211    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:43:56.011882 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:56.011847    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:56.875754 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:56.875722    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:57.053311 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.053273    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:43:57.243349 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.243253    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc"]
Nov 01 06:43:57.243881 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.243865    5619 cpu_manager.go:414] "containerMap: removing container" podUID="304878ad-a41a-4349-8ec7-3c7d38f4efbc" containerName="maas-api"
Nov 01 06:43:57.243938 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.243882    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="304878ad-a41a-4349-8ec7-3c7d38f4efbc" containerName="maas-api"
Nov 01 06:43:57.243938 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.243904    5619 cpu_manager.go:414] "containerMap: removing container" podUID="9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" containerName="istio-proxy"
Nov 01 06:43:57.243938 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.243910    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" containerName="istio-proxy"
Nov 01 06:43:57.243938 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.243924    5619 cpu_manager.go:414] "containerMap: removing container" podUID="81cdf4f9-7e3c-47c0-b669-e06ec353fdf6" containerName="main"
Nov 01 06:43:57.243938 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.243930    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81cdf4f9-7e3c-47c0-b669-e06ec353fdf6" containerName="main"
Nov 01 06:43:57.244127 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.244033    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="81cdf4f9-7e3c-47c0-b669-e06ec353fdf6" containerName="main"
Nov 01 06:43:57.244127 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.244048    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ff3bbf5-47d7-4a26-95c6-ed22acc794c3" containerName="istio-proxy"
Nov 01 06:43:57.248672 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.248652    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc"
Nov 01 06:43:57.305246 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.305216    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc"]
Nov 01 06:43:57.379604 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.379568    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-5be4fa\""
Nov 01 06:43:57.379842 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.379642    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-5be4fa-dockercfg-b5ngj\""
Nov 01 06:43:57.396341 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.394544    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e57167-aba3-406a-9a21-a48ae4584959-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc\" (UID: \"d3e57167-aba3-406a-9a21-a48ae4584959\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc"
Nov 01 06:43:57.396341 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.394608    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnd2d\" (UniqueName: \"kubernetes.io/projected/d3e57167-aba3-406a-9a21-a48ae4584959-kube-api-access-gnd2d\") pod \"payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc\" (UID: \"d3e57167-aba3-406a-9a21-a48ae4584959\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc"
Nov 01 06:43:57.424934 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.424903    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns"]
Nov 01 06:43:57.428843 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.428826    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns"
Nov 01 06:43:57.461018 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.460989    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns"]
Nov 01 06:43:57.495711 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.495642    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e57167-aba3-406a-9a21-a48ae4584959-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc\" (UID: \"d3e57167-aba3-406a-9a21-a48ae4584959\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc"
Nov 01 06:43:57.495711 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.495696    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnd2d\" (UniqueName: \"kubernetes.io/projected/d3e57167-aba3-406a-9a21-a48ae4584959-kube-api-access-gnd2d\") pod \"payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc\" (UID: \"d3e57167-aba3-406a-9a21-a48ae4584959\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc"
Nov 01 06:43:57.496494 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.496470    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e57167-aba3-406a-9a21-a48ae4584959-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc\" (UID: \"d3e57167-aba3-406a-9a21-a48ae4584959\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc"
Nov 01 06:43:57.557771 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.557738    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnd2d\" (UniqueName: \"kubernetes.io/projected/d3e57167-aba3-406a-9a21-a48ae4584959-kube-api-access-gnd2d\") pod \"payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc\" (UID: \"d3e57167-aba3-406a-9a21-a48ae4584959\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc"
Nov 01 06:43:57.562548 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.562520    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc"
Nov 01 06:43:57.597366 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.597334    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4wdr\" (UniqueName: \"kubernetes.io/projected/22cb8e7f-72a8-4cf7-a419-8470c27d8799-kube-api-access-d4wdr\") pod \"payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns\" (UID: \"22cb8e7f-72a8-4cf7-a419-8470c27d8799\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns"
Nov 01 06:43:57.597521 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.597376    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22cb8e7f-72a8-4cf7-a419-8470c27d8799-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns\" (UID: \"22cb8e7f-72a8-4cf7-a419-8470c27d8799\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns"
Nov 01 06:43:57.699386 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.699340    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4wdr\" (UniqueName: \"kubernetes.io/projected/22cb8e7f-72a8-4cf7-a419-8470c27d8799-kube-api-access-d4wdr\") pod \"payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns\" (UID: \"22cb8e7f-72a8-4cf7-a419-8470c27d8799\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns"
Nov 01 06:43:57.699562 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.699393    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22cb8e7f-72a8-4cf7-a419-8470c27d8799-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns\" (UID: \"22cb8e7f-72a8-4cf7-a419-8470c27d8799\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns"
Nov 01 06:43:57.700168 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.700146    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22cb8e7f-72a8-4cf7-a419-8470c27d8799-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns\" (UID: \"22cb8e7f-72a8-4cf7-a419-8470c27d8799\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns"
Nov 01 06:43:57.761726 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.761648    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4wdr\" (UniqueName: \"kubernetes.io/projected/22cb8e7f-72a8-4cf7-a419-8470c27d8799-kube-api-access-d4wdr\") pod \"payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns\" (UID: \"22cb8e7f-72a8-4cf7-a419-8470c27d8799\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns"
Nov 01 06:43:57.936253 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:57.936225    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc"]
Nov 01 06:43:57.938524 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:43:57.937439    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e57167_aba3_406a_9a21_a48ae4584959.slice/crio-4e87e00161b0bbcc506a4ead64895637af71e9947b42c3a7f18665cea339b126 WatchSource:0}: Error finding container 4e87e00161b0bbcc506a4ead64895637af71e9947b42c3a7f18665cea339b126: Status 404 returned error can't find the container with id 4e87e00161b0bbcc506a4ead64895637af71e9947b42c3a7f18665cea339b126
Nov 01 06:43:58.042878 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:58.042814    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns"
Nov 01 06:43:58.415697 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:58.415667    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns"]
Nov 01 06:43:58.417208 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:43:58.417185    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22cb8e7f_72a8_4cf7_a419_8470c27d8799.slice/crio-491e3c3b13386e8e1b430dceddf88873ec02db7d4bf2c9c46f3c6bc805972903 WatchSource:0}: Error finding container 491e3c3b13386e8e1b430dceddf88873ec02db7d4bf2c9c46f3c6bc805972903: Status 404 returned error can't find the container with id 491e3c3b13386e8e1b430dceddf88873ec02db7d4bf2c9c46f3c6bc805972903
Nov 01 06:43:58.924407 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:58.924371    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc" event={"ID":"d3e57167-aba3-406a-9a21-a48ae4584959","Type":"ContainerStarted","Data":"4ccceef3d34c6cdaff340f6549499e84dffe02e8b654bc2477c7eff905d2c2a3"}
Nov 01 06:43:58.924597 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:58.924439    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc"
Nov 01 06:43:58.924597 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:58.924458    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc" event={"ID":"d3e57167-aba3-406a-9a21-a48ae4584959","Type":"ContainerStarted","Data":"4e87e00161b0bbcc506a4ead64895637af71e9947b42c3a7f18665cea339b126"}
Nov 01 06:43:58.925920 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:58.925890    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns" event={"ID":"22cb8e7f-72a8-4cf7-a419-8470c27d8799","Type":"ContainerStarted","Data":"9b0e1c40dd88bc5942d60a95e52a2ca35f0f466a6d95f22f210868675a8e9c93"}
Nov 01 06:43:58.926037 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:58.925940    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns"
Nov 01 06:43:58.926037 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:58.925957    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns" event={"ID":"22cb8e7f-72a8-4cf7-a419-8470c27d8799","Type":"ContainerStarted","Data":"491e3c3b13386e8e1b430dceddf88873ec02db7d4bf2c9c46f3c6bc805972903"}
Nov 01 06:43:59.041595 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:43:59.041545    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc" podStartSLOduration=2.041527678 podStartE2EDuration="2.041527678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:43:59.040602144 +0000 UTC m=+3476.251425653" watchObservedRunningTime="2026-11-01 06:43:59.041527678 +0000 UTC m=+3476.252351188"
Nov 01 06:44:00.745011 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:00.744971    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn" podUID="623f7715-a2cc-41d3-813e-236c25f7925d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:44:00.961007 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:00.960971    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:01.199368 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:01.199287    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:01.349902 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:01.349206    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:02.306261 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:02.306200    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns" podStartSLOduration=5.306186906 podStartE2EDuration="5.306186906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:43:59.161639572 +0000 UTC m=+3476.372463082" watchObservedRunningTime="2026-11-01 06:44:02.306186906 +0000 UTC m=+3479.517010464"
Nov 01 06:44:02.306806 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:02.306646    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"]
Nov 01 06:44:02.307993 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:02.307965    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn" podUID="623f7715-a2cc-41d3-813e-236c25f7925d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:44:02.357378 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:02.357336    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"]
Nov 01 06:44:02.361622 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:02.361595    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"
Nov 01 06:44:02.423494 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:02.423458    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"]
Nov 01 06:44:02.459045 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:02.459006    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da368949-f806-4f1d-81e0-0bb8b6318eea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc\" (UID: \"da368949-f806-4f1d-81e0-0bb8b6318eea\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"
Nov 01 06:44:02.459493 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:02.459465    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xcq5\" (UniqueName: \"kubernetes.io/projected/da368949-f806-4f1d-81e0-0bb8b6318eea-kube-api-access-9xcq5\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc\" (UID: \"da368949-f806-4f1d-81e0-0bb8b6318eea\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"
Nov 01 06:44:02.459702 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:02.459680    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/da368949-f806-4f1d-81e0-0bb8b6318eea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc\" (UID: \"da368949-f806-4f1d-81e0-0bb8b6318eea\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"
Nov 01 06:44:02.459942 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:02.459922    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/da368949-f806-4f1d-81e0-0bb8b6318eea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc\" (UID: \"da368949-f806-4f1d-81e0-0bb8b6318eea\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"
Nov 01 06:44:02.561705 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:02.561598    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/da368949-f806-4f1d-81e0-0bb8b6318eea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc\" (UID: \"da368949-f806-4f1d-81e0-0bb8b6318eea\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"
Nov 01 06:44:02.561904 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:02.561743    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da368949-f806-4f1d-81e0-0bb8b6318eea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc\" (UID: \"da368949-f806-4f1d-81e0-0bb8b6318eea\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"
Nov 01 06:44:02.561904 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:02.561775    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xcq5\" (UniqueName: \"kubernetes.io/projected/da368949-f806-4f1d-81e0-0bb8b6318eea-kube-api-access-9xcq5\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc\" (UID: \"da368949-f806-4f1d-81e0-0bb8b6318eea\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"
Nov 01 06:44:02.561904 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:02.561807    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/da368949-f806-4f1d-81e0-0bb8b6318eea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc\" (UID: \"da368949-f806-4f1d-81e0-0bb8b6318eea\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"
Nov 01 06:44:02.562473 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:02.562452    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/da368949-f806-4f1d-81e0-0bb8b6318eea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc\" (UID: \"da368949-f806-4f1d-81e0-0bb8b6318eea\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"
Nov 01 06:44:02.562620 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:02.562599    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da368949-f806-4f1d-81e0-0bb8b6318eea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc\" (UID: \"da368949-f806-4f1d-81e0-0bb8b6318eea\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"
Nov 01 06:44:02.564469 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:02.564440    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/da368949-f806-4f1d-81e0-0bb8b6318eea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc\" (UID: \"da368949-f806-4f1d-81e0-0bb8b6318eea\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"
Nov 01 06:44:02.623937 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:02.623904    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xcq5\" (UniqueName: \"kubernetes.io/projected/da368949-f806-4f1d-81e0-0bb8b6318eea-kube-api-access-9xcq5\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc\" (UID: \"da368949-f806-4f1d-81e0-0bb8b6318eea\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"
Nov 01 06:44:02.676344 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:02.676289    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"
Nov 01 06:44:03.054610 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:03.054581    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"]
Nov 01 06:44:03.055736 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:44:03.055708    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda368949_f806_4f1d_81e0_0bb8b6318eea.slice/crio-e41b8a01b6b939395953d15fdf1aa615186d352085d14caaddd6e85b8c97c272 WatchSource:0}: Error finding container e41b8a01b6b939395953d15fdf1aa615186d352085d14caaddd6e85b8c97c272: Status 404 returned error can't find the container with id e41b8a01b6b939395953d15fdf1aa615186d352085d14caaddd6e85b8c97c272
Nov 01 06:44:03.959441 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:03.959398    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc" event={"ID":"da368949-f806-4f1d-81e0-0bb8b6318eea","Type":"ContainerStarted","Data":"ea88484bac5d91975290e825402ec4cc2082c9796145c24ddec1280591fef85e"}
Nov 01 06:44:03.959882 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:03.959453    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc" event={"ID":"da368949-f806-4f1d-81e0-0bb8b6318eea","Type":"ContainerStarted","Data":"e41b8a01b6b939395953d15fdf1aa615186d352085d14caaddd6e85b8c97c272"}
Nov 01 06:44:03.959882 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:03.959539    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"
Nov 01 06:44:04.081998 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:04.081944    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc" podStartSLOduration=2.08192923 podStartE2EDuration="2.08192923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:44:04.080763293 +0000 UTC m=+3481.291586803" watchObservedRunningTime="2026-11-01 06:44:04.08192923 +0000 UTC m=+3481.292752736"
Nov 01 06:44:05.667659 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:05.667626    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:07.307583 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:07.307524    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn" podUID="623f7715-a2cc-41d3-813e-236c25f7925d" containerName="payload-processing" containerID="cri-o://79046f108a85eb1efeb60ba1c41cab8ca745f4516bf7577b5929660f586f05cd" gracePeriod=25
Nov 01 06:44:07.570149 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:07.570073    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"
Nov 01 06:44:07.631869 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:07.631842    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/623f7715-a2cc-41d3-813e-236c25f7925d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/623f7715-a2cc-41d3-813e-236c25f7925d-routing-overlay-volume\") pod \"623f7715-a2cc-41d3-813e-236c25f7925d\" (UID: \"623f7715-a2cc-41d3-813e-236c25f7925d\") "
Nov 01 06:44:07.631985 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:07.631881    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/623f7715-a2cc-41d3-813e-236c25f7925d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/623f7715-a2cc-41d3-813e-236c25f7925d-provider-credentials\") pod \"623f7715-a2cc-41d3-813e-236c25f7925d\" (UID: \"623f7715-a2cc-41d3-813e-236c25f7925d\") "
Nov 01 06:44:07.631985 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:07.631945    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/623f7715-a2cc-41d3-813e-236c25f7925d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/623f7715-a2cc-41d3-813e-236c25f7925d-plugins-config-volume\") pod \"623f7715-a2cc-41d3-813e-236c25f7925d\" (UID: \"623f7715-a2cc-41d3-813e-236c25f7925d\") "
Nov 01 06:44:07.632094 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:07.632064    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/623f7715-a2cc-41d3-813e-236c25f7925d-kube-api-access-vm9lh\" (UniqueName: \"kubernetes.io/projected/623f7715-a2cc-41d3-813e-236c25f7925d-kube-api-access-vm9lh\") pod \"623f7715-a2cc-41d3-813e-236c25f7925d\" (UID: \"623f7715-a2cc-41d3-813e-236c25f7925d\") "
Nov 01 06:44:07.632261 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:07.632236    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/623f7715-a2cc-41d3-813e-236c25f7925d-routing-overlay-volume" pod "623f7715-a2cc-41d3-813e-236c25f7925d" (UID: "623f7715-a2cc-41d3-813e-236c25f7925d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:44:07.632405 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:07.632377    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/623f7715-a2cc-41d3-813e-236c25f7925d-plugins-config-volume" pod "623f7715-a2cc-41d3-813e-236c25f7925d" (UID: "623f7715-a2cc-41d3-813e-236c25f7925d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:44:07.632576 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:07.632558    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/623f7715-a2cc-41d3-813e-236c25f7925d-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:07.632626 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:07.632583    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/623f7715-a2cc-41d3-813e-236c25f7925d-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:07.633981 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:07.633956    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623f7715-a2cc-41d3-813e-236c25f7925d-provider-credentials" pod "623f7715-a2cc-41d3-813e-236c25f7925d" (UID: "623f7715-a2cc-41d3-813e-236c25f7925d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:44:07.634078 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:07.633995    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623f7715-a2cc-41d3-813e-236c25f7925d-kube-api-access-vm9lh" pod "623f7715-a2cc-41d3-813e-236c25f7925d" (UID: "623f7715-a2cc-41d3-813e-236c25f7925d"). InnerVolumeSpecName "kube-api-access-vm9lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:44:07.733690 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:07.733657    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm9lh\" (UniqueName: \"kubernetes.io/projected/623f7715-a2cc-41d3-813e-236c25f7925d-kube-api-access-vm9lh\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:07.733690 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:07.733687    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/623f7715-a2cc-41d3-813e-236c25f7925d-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:07.985706 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:07.985669    5619 generic.go:356] "Generic (PLEG): container finished" podID="623f7715-a2cc-41d3-813e-236c25f7925d" containerID="79046f108a85eb1efeb60ba1c41cab8ca745f4516bf7577b5929660f586f05cd" exitCode=0
Nov 01 06:44:07.985706 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:07.985715    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"
Nov 01 06:44:07.985706 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:07.985720    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn" event={"ID":"623f7715-a2cc-41d3-813e-236c25f7925d","Type":"ContainerDied","Data":"79046f108a85eb1efeb60ba1c41cab8ca745f4516bf7577b5929660f586f05cd"}
Nov 01 06:44:07.986018 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:07.985761    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn" event={"ID":"623f7715-a2cc-41d3-813e-236c25f7925d","Type":"ContainerDied","Data":"19cf23ae462e973f934323584ed1468406fb62a9ae306a6d937f7e65efb11101"}
Nov 01 06:44:07.986018 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:07.985786    5619 scope.go:122] "RemoveContainer" containerID="79046f108a85eb1efeb60ba1c41cab8ca745f4516bf7577b5929660f586f05cd"
Nov 01 06:44:07.998505 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:07.998485    5619 scope.go:122] "RemoveContainer" containerID="79046f108a85eb1efeb60ba1c41cab8ca745f4516bf7577b5929660f586f05cd"
Nov 01 06:44:07.998793 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:07.998769    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79046f108a85eb1efeb60ba1c41cab8ca745f4516bf7577b5929660f586f05cd\": container with ID starting with 79046f108a85eb1efeb60ba1c41cab8ca745f4516bf7577b5929660f586f05cd not found: ID does not exist" containerID="79046f108a85eb1efeb60ba1c41cab8ca745f4516bf7577b5929660f586f05cd"
Nov 01 06:44:07.998920 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:07.998800    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79046f108a85eb1efeb60ba1c41cab8ca745f4516bf7577b5929660f586f05cd"} err="failed to get container status \"79046f108a85eb1efeb60ba1c41cab8ca745f4516bf7577b5929660f586f05cd\": rpc error: code = NotFound desc = could not find container \"79046f108a85eb1efeb60ba1c41cab8ca745f4516bf7577b5929660f586f05cd\": container with ID starting with 79046f108a85eb1efeb60ba1c41cab8ca745f4516bf7577b5929660f586f05cd not found: ID does not exist"
Nov 01 06:44:08.153145 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:08.153114    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"]
Nov 01 06:44:08.158847 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:08.158810    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-9598l7jnn"]
Nov 01 06:44:08.963208 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:08.963173    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="623f7715-a2cc-41d3-813e-236c25f7925d" path="/var/lib/kubelet/pods/623f7715-a2cc-41d3-813e-236c25f7925d/volumes"
Nov 01 06:44:09.935645 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:09.935617    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns"
Nov 01 06:44:09.935841 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:09.935683    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc"
Nov 01 06:44:10.378858 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:10.378817    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:12.619890 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:12.619850    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:12.814829 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:12.814792    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:13.744535 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:13.744500    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"]
Nov 01 06:44:13.747649 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:13.747615    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc" podUID="da368949-f806-4f1d-81e0-0bb8b6318eea" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:44:15.154822 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:15.154786    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:16.788585 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:16.788546    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:18.559886 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:18.559849    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:18.745169 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:18.745130    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc" podUID="da368949-f806-4f1d-81e0-0bb8b6318eea" containerName="payload-processing" containerID="cri-o://ea88484bac5d91975290e825402ec4cc2082c9796145c24ddec1280591fef85e" gracePeriod=25
Nov 01 06:44:19.036960 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.036932    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"
Nov 01 06:44:19.054795 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.054762    5619 generic.go:356] "Generic (PLEG): container finished" podID="da368949-f806-4f1d-81e0-0bb8b6318eea" containerID="ea88484bac5d91975290e825402ec4cc2082c9796145c24ddec1280591fef85e" exitCode=0
Nov 01 06:44:19.054956 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.054812    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"
Nov 01 06:44:19.054956 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.054811    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc" event={"ID":"da368949-f806-4f1d-81e0-0bb8b6318eea","Type":"ContainerDied","Data":"ea88484bac5d91975290e825402ec4cc2082c9796145c24ddec1280591fef85e"}
Nov 01 06:44:19.054956 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.054878    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc" event={"ID":"da368949-f806-4f1d-81e0-0bb8b6318eea","Type":"ContainerDied","Data":"e41b8a01b6b939395953d15fdf1aa615186d352085d14caaddd6e85b8c97c272"}
Nov 01 06:44:19.054956 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.054906    5619 scope.go:122] "RemoveContainer" containerID="ea88484bac5d91975290e825402ec4cc2082c9796145c24ddec1280591fef85e"
Nov 01 06:44:19.063968 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.063950    5619 scope.go:122] "RemoveContainer" containerID="ea88484bac5d91975290e825402ec4cc2082c9796145c24ddec1280591fef85e"
Nov 01 06:44:19.064273 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:19.064249    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea88484bac5d91975290e825402ec4cc2082c9796145c24ddec1280591fef85e\": container with ID starting with ea88484bac5d91975290e825402ec4cc2082c9796145c24ddec1280591fef85e not found: ID does not exist" containerID="ea88484bac5d91975290e825402ec4cc2082c9796145c24ddec1280591fef85e"
Nov 01 06:44:19.064367 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.064280    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea88484bac5d91975290e825402ec4cc2082c9796145c24ddec1280591fef85e"} err="failed to get container status \"ea88484bac5d91975290e825402ec4cc2082c9796145c24ddec1280591fef85e\": rpc error: code = NotFound desc = could not find container \"ea88484bac5d91975290e825402ec4cc2082c9796145c24ddec1280591fef85e\": container with ID starting with ea88484bac5d91975290e825402ec4cc2082c9796145c24ddec1280591fef85e not found: ID does not exist"
Nov 01 06:44:19.178516 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.178108    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da368949-f806-4f1d-81e0-0bb8b6318eea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da368949-f806-4f1d-81e0-0bb8b6318eea-plugins-config-volume\") pod \"da368949-f806-4f1d-81e0-0bb8b6318eea\" (UID: \"da368949-f806-4f1d-81e0-0bb8b6318eea\") "
Nov 01 06:44:19.178516 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.178219    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da368949-f806-4f1d-81e0-0bb8b6318eea-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/da368949-f806-4f1d-81e0-0bb8b6318eea-routing-overlay-volume\") pod \"da368949-f806-4f1d-81e0-0bb8b6318eea\" (UID: \"da368949-f806-4f1d-81e0-0bb8b6318eea\") "
Nov 01 06:44:19.178516 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.178411    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da368949-f806-4f1d-81e0-0bb8b6318eea-kube-api-access-9xcq5\" (UniqueName: \"kubernetes.io/projected/da368949-f806-4f1d-81e0-0bb8b6318eea-kube-api-access-9xcq5\") pod \"da368949-f806-4f1d-81e0-0bb8b6318eea\" (UID: \"da368949-f806-4f1d-81e0-0bb8b6318eea\") "
Nov 01 06:44:19.178877 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.178563    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da368949-f806-4f1d-81e0-0bb8b6318eea-plugins-config-volume" pod "da368949-f806-4f1d-81e0-0bb8b6318eea" (UID: "da368949-f806-4f1d-81e0-0bb8b6318eea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:44:19.178877 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.178659    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da368949-f806-4f1d-81e0-0bb8b6318eea-provider-credentials\" (UniqueName: \"kubernetes.io/projected/da368949-f806-4f1d-81e0-0bb8b6318eea-provider-credentials\") pod \"da368949-f806-4f1d-81e0-0bb8b6318eea\" (UID: \"da368949-f806-4f1d-81e0-0bb8b6318eea\") "
Nov 01 06:44:19.178995 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.178900    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da368949-f806-4f1d-81e0-0bb8b6318eea-routing-overlay-volume" pod "da368949-f806-4f1d-81e0-0bb8b6318eea" (UID: "da368949-f806-4f1d-81e0-0bb8b6318eea"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:44:19.179215 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.179191    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da368949-f806-4f1d-81e0-0bb8b6318eea-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:19.179292 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.179221    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/da368949-f806-4f1d-81e0-0bb8b6318eea-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:19.181452 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.181412    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da368949-f806-4f1d-81e0-0bb8b6318eea-provider-credentials" pod "da368949-f806-4f1d-81e0-0bb8b6318eea" (UID: "da368949-f806-4f1d-81e0-0bb8b6318eea"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:44:19.181574 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.181552    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da368949-f806-4f1d-81e0-0bb8b6318eea-kube-api-access-9xcq5" pod "da368949-f806-4f1d-81e0-0bb8b6318eea" (UID: "da368949-f806-4f1d-81e0-0bb8b6318eea"). InnerVolumeSpecName "kube-api-access-9xcq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:44:19.280812 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.280771    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xcq5\" (UniqueName: \"kubernetes.io/projected/da368949-f806-4f1d-81e0-0bb8b6318eea-kube-api-access-9xcq5\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:19.280991 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.280813    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/da368949-f806-4f1d-81e0-0bb8b6318eea-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:19.478509 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.478420    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"]
Nov 01 06:44:19.482011 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.481987    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-6949rmvkc"]
Nov 01 06:44:19.857725 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:19.857692    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:20.961758 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:20.961728    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da368949-f806-4f1d-81e0-0bb8b6318eea" path="/var/lib/kubelet/pods/da368949-f806-4f1d-81e0-0bb8b6318eea/volumes"
Nov 01 06:44:22.748461 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:22.748425    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:24.536472 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:24.536436    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:25.880872 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:25.880830    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:44:26.786893 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:26.786857    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:26.929198 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:26.929161    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:28.046021 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.045991    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"]
Nov 01 06:44:28.046628 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.046609    5619 cpu_manager.go:414] "containerMap: removing container" podUID="da368949-f806-4f1d-81e0-0bb8b6318eea" containerName="payload-processing"
Nov 01 06:44:28.046680 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.046631    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da368949-f806-4f1d-81e0-0bb8b6318eea" containerName="payload-processing"
Nov 01 06:44:28.046680 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.046668    5619 cpu_manager.go:414] "containerMap: removing container" podUID="623f7715-a2cc-41d3-813e-236c25f7925d" containerName="payload-processing"
Nov 01 06:44:28.046680 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.046673    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="623f7715-a2cc-41d3-813e-236c25f7925d" containerName="payload-processing"
Nov 01 06:44:28.046785 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.046770    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="623f7715-a2cc-41d3-813e-236c25f7925d" containerName="payload-processing"
Nov 01 06:44:28.046823 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.046787    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="da368949-f806-4f1d-81e0-0bb8b6318eea" containerName="payload-processing"
Nov 01 06:44:28.051467 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.051446    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"
Nov 01 06:44:28.159563 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.159519    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"]
Nov 01 06:44:28.177244 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.177214    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg4wf\" (UniqueName: \"kubernetes.io/projected/7214a378-e37c-4625-b431-00f94c30799e-kube-api-access-rg4wf\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl\" (UID: \"7214a378-e37c-4625-b431-00f94c30799e\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"
Nov 01 06:44:28.177436 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.177408    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7214a378-e37c-4625-b431-00f94c30799e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl\" (UID: \"7214a378-e37c-4625-b431-00f94c30799e\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"
Nov 01 06:44:28.177498 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.177487    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7214a378-e37c-4625-b431-00f94c30799e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl\" (UID: \"7214a378-e37c-4625-b431-00f94c30799e\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"
Nov 01 06:44:28.177543 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.177531    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7214a378-e37c-4625-b431-00f94c30799e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl\" (UID: \"7214a378-e37c-4625-b431-00f94c30799e\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"
Nov 01 06:44:28.278378 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.278343    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7214a378-e37c-4625-b431-00f94c30799e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl\" (UID: \"7214a378-e37c-4625-b431-00f94c30799e\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"
Nov 01 06:44:28.278537 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.278409    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7214a378-e37c-4625-b431-00f94c30799e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl\" (UID: \"7214a378-e37c-4625-b431-00f94c30799e\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"
Nov 01 06:44:28.278537 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.278457    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7214a378-e37c-4625-b431-00f94c30799e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl\" (UID: \"7214a378-e37c-4625-b431-00f94c30799e\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"
Nov 01 06:44:28.278646 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.278565    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg4wf\" (UniqueName: \"kubernetes.io/projected/7214a378-e37c-4625-b431-00f94c30799e-kube-api-access-rg4wf\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl\" (UID: \"7214a378-e37c-4625-b431-00f94c30799e\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"
Nov 01 06:44:28.279286 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.279254    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7214a378-e37c-4625-b431-00f94c30799e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl\" (UID: \"7214a378-e37c-4625-b431-00f94c30799e\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"
Nov 01 06:44:28.279411 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.279258    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7214a378-e37c-4625-b431-00f94c30799e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl\" (UID: \"7214a378-e37c-4625-b431-00f94c30799e\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"
Nov 01 06:44:28.280844 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.280823    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7214a378-e37c-4625-b431-00f94c30799e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl\" (UID: \"7214a378-e37c-4625-b431-00f94c30799e\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"
Nov 01 06:44:28.340140 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.340117    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg4wf\" (UniqueName: \"kubernetes.io/projected/7214a378-e37c-4625-b431-00f94c30799e-kube-api-access-rg4wf\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl\" (UID: \"7214a378-e37c-4625-b431-00f94c30799e\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"
Nov 01 06:44:28.364603 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.364574    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"
Nov 01 06:44:28.749180 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:28.749150    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"]
Nov 01 06:44:28.750888 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:44:28.750846    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7214a378_e37c_4625_b431_00f94c30799e.slice/crio-c5b7081ffd5fdcbf74d671ba4b443df647557bb5d2c81840efb082212f8fd5fd WatchSource:0}: Error finding container c5b7081ffd5fdcbf74d671ba4b443df647557bb5d2c81840efb082212f8fd5fd: Status 404 returned error can't find the container with id c5b7081ffd5fdcbf74d671ba4b443df647557bb5d2c81840efb082212f8fd5fd
Nov 01 06:44:29.117032 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:29.116999    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl" event={"ID":"7214a378-e37c-4625-b431-00f94c30799e","Type":"ContainerStarted","Data":"450b15e2d4ee7cd09d7480f597862e447a3d4c345077ebd1d1fcd50f73686711"}
Nov 01 06:44:29.117486 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:29.117056    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"
Nov 01 06:44:29.117486 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:29.117074    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl" event={"ID":"7214a378-e37c-4625-b431-00f94c30799e","Type":"ContainerStarted","Data":"c5b7081ffd5fdcbf74d671ba4b443df647557bb5d2c81840efb082212f8fd5fd"}
Nov 01 06:44:29.237156 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:29.237107    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl" podStartSLOduration=1.237092374 podStartE2EDuration="1.237092374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:44:29.2345956 +0000 UTC m=+3506.445419103" watchObservedRunningTime="2026-11-01 06:44:29.237092374 +0000 UTC m=+3506.447915884"
Nov 01 06:44:29.349228 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:29.349186    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:31.109010 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:31.108980    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-089523cf8a24: secret "agc-cred-089523cf8a24" not found
Nov 01 06:44:31.109010 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:31.109008    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-31d2d9550e36: secret "agc-cred-31d2d9550e36" not found
Nov 01 06:44:31.109010 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:31.109016    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-a97b9b6eb74d: secret "agc-cred-a97b9b6eb74d" not found
Nov 01 06:44:31.109586 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:31.109032    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-e14e279743a3: secret "agc-cred-e14e279743a3" not found
Nov 01 06:44:31.109586 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:31.109048    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9: [secret "agc-cred-089523cf8a24" not found, secret "agc-cred-31d2d9550e36" not found, secret "agc-cred-a97b9b6eb74d" not found, secret "agc-cred-e14e279743a3" not found]
Nov 01 06:44:31.109586 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:31.109145    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-provider-credentials podName:7afa361c-dc3c-4281-b43b-c7d4a7dfcac1 nodeName:}" failed. No retries permitted until 2026-11-01 06:44:31.609124951 +0000 UTC m=+3508.819948441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9" (UID: "7afa361c-dc3c-4281-b43b-c7d4a7dfcac1") : [secret "agc-cred-089523cf8a24" not found, secret "agc-cred-31d2d9550e36" not found, secret "agc-cred-a97b9b6eb74d" not found, secret "agc-cred-e14e279743a3" not found]
Nov 01 06:44:31.615157 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:31.615126    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-089523cf8a24: secret "agc-cred-089523cf8a24" not found
Nov 01 06:44:31.615157 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:31.615157    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-31d2d9550e36: secret "agc-cred-31d2d9550e36" not found
Nov 01 06:44:31.615378 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:31.615166    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-a97b9b6eb74d: secret "agc-cred-a97b9b6eb74d" not found
Nov 01 06:44:31.615378 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:31.615173    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-e14e279743a3: secret "agc-cred-e14e279743a3" not found
Nov 01 06:44:31.615378 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:31.615188    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9: [secret "agc-cred-089523cf8a24" not found, secret "agc-cred-31d2d9550e36" not found, secret "agc-cred-a97b9b6eb74d" not found, secret "agc-cred-e14e279743a3" not found]
Nov 01 06:44:31.615378 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:31.615249    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-provider-credentials podName:7afa361c-dc3c-4281-b43b-c7d4a7dfcac1 nodeName:}" failed. No retries permitted until 2026-11-01 06:44:32.615232647 +0000 UTC m=+3509.826056151 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9" (UID: "7afa361c-dc3c-4281-b43b-c7d4a7dfcac1") : [secret "agc-cred-089523cf8a24" not found, secret "agc-cred-31d2d9550e36" not found, secret "agc-cred-a97b9b6eb74d" not found, secret "agc-cred-e14e279743a3" not found]
Nov 01 06:44:31.951565 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:31.951481    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:32.626885 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:32.626844    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-089523cf8a24: secret "agc-cred-089523cf8a24" not found
Nov 01 06:44:32.626885 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:32.626884    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-31d2d9550e36: secret "agc-cred-31d2d9550e36" not found
Nov 01 06:44:32.626885 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:32.626895    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-a97b9b6eb74d: secret "agc-cred-a97b9b6eb74d" not found
Nov 01 06:44:32.627388 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:32.626902    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-e14e279743a3: secret "agc-cred-e14e279743a3" not found
Nov 01 06:44:32.627388 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:32.626917    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9: [secret "agc-cred-089523cf8a24" not found, secret "agc-cred-31d2d9550e36" not found, secret "agc-cred-a97b9b6eb74d" not found, secret "agc-cred-e14e279743a3" not found]
Nov 01 06:44:32.627388 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:32.626977    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-provider-credentials podName:7afa361c-dc3c-4281-b43b-c7d4a7dfcac1 nodeName:}" failed. No retries permitted until 2026-11-01 06:44:34.626961475 +0000 UTC m=+3511.837784962 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9" (UID: "7afa361c-dc3c-4281-b43b-c7d4a7dfcac1") : [secret "agc-cred-089523cf8a24" not found, secret "agc-cred-31d2d9550e36" not found, secret "agc-cred-a97b9b6eb74d" not found, secret "agc-cred-e14e279743a3" not found]
Nov 01 06:44:34.071306 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:34.071268    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:34.648201 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:34.648170    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-089523cf8a24: secret "agc-cred-089523cf8a24" not found
Nov 01 06:44:34.648201 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:34.648197    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-31d2d9550e36: secret "agc-cred-31d2d9550e36" not found
Nov 01 06:44:34.648201 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:34.648207    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-a97b9b6eb74d: secret "agc-cred-a97b9b6eb74d" not found
Nov 01 06:44:34.648201 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:34.648213    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-e14e279743a3: secret "agc-cred-e14e279743a3" not found
Nov 01 06:44:34.648499 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:34.648227    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9: [secret "agc-cred-089523cf8a24" not found, secret "agc-cred-31d2d9550e36" not found, secret "agc-cred-a97b9b6eb74d" not found, secret "agc-cred-e14e279743a3" not found]
Nov 01 06:44:34.648499 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:34.648280    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-provider-credentials podName:7afa361c-dc3c-4281-b43b-c7d4a7dfcac1 nodeName:}" failed. No retries permitted until 2026-11-01 06:44:38.648265671 +0000 UTC m=+3515.859089172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9" (UID: "7afa361c-dc3c-4281-b43b-c7d4a7dfcac1") : [secret "agc-cred-089523cf8a24" not found, secret "agc-cred-31d2d9550e36" not found, secret "agc-cred-a97b9b6eb74d" not found, secret "agc-cred-e14e279743a3" not found]
Nov 01 06:44:36.357941 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:36.357903    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:38.696022 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:38.695983    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-089523cf8a24: secret "agc-cred-089523cf8a24" not found
Nov 01 06:44:38.696022 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:38.696019    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-31d2d9550e36: secret "agc-cred-31d2d9550e36" not found
Nov 01 06:44:38.696022 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:38.696028    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-a97b9b6eb74d: secret "agc-cred-a97b9b6eb74d" not found
Nov 01 06:44:38.696613 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:38.696037    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-e14e279743a3: secret "agc-cred-e14e279743a3" not found
Nov 01 06:44:38.696613 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:38.696054    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9: [secret "agc-cred-089523cf8a24" not found, secret "agc-cred-31d2d9550e36" not found, secret "agc-cred-a97b9b6eb74d" not found, secret "agc-cred-e14e279743a3" not found]
Nov 01 06:44:38.696613 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:38.696143    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-provider-credentials podName:7afa361c-dc3c-4281-b43b-c7d4a7dfcac1 nodeName:}" failed. No retries permitted until 2026-11-01 06:44:46.696122147 +0000 UTC m=+3523.906945640 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9" (UID: "7afa361c-dc3c-4281-b43b-c7d4a7dfcac1") : [secret "agc-cred-089523cf8a24" not found, secret "agc-cred-31d2d9550e36" not found, secret "agc-cred-a97b9b6eb74d" not found, secret "agc-cred-e14e279743a3" not found]
Nov 01 06:44:38.860218 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:38.860181    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:39.768953 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:39.768914    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"]
Nov 01 06:44:39.771645 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:39.771608    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl" podUID="7214a378-e37c-4625-b431-00f94c30799e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:44:39.819617 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:39.819586    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"]
Nov 01 06:44:39.855935 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:39.855905    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:40.049274 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:40.049187    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:41.236669 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:41.236583    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:41.665782 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:41.665743    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:42.923093 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:42.923046    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:43.062036 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:43.062000    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:43.718656 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:43.718624    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:43.910810 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:43.910763    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:44.769935 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:44.769895    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl" podUID="7214a378-e37c-4625-b431-00f94c30799e" containerName="payload-processing" containerID="cri-o://450b15e2d4ee7cd09d7480f597862e447a3d4c345077ebd1d1fcd50f73686711" gracePeriod=25
Nov 01 06:44:44.820460 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:44.820423    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9" podUID="7afa361c-dc3c-4281-b43b-c7d4a7dfcac1" containerName="payload-processing" containerID="cri-o://3c5fdc6903820c5a2db272f4dc1abede3138f5c423db3dc85481e0a5afa74a16" gracePeriod=25
Nov 01 06:44:44.886524 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:44.886498    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"]
Nov 01 06:44:44.890531 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:44.890512    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"
Nov 01 06:44:44.946258 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:44.946230    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"agc-cred-47cacd68a67a\""
Nov 01 06:44:44.986817 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:44.986750    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nm5r\" (UniqueName: \"kubernetes.io/projected/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-kube-api-access-2nm5r\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44\" (UID: \"c2eb9535-5242-4e53-af4a-3c03c5f56ff4\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"
Nov 01 06:44:44.987003 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:44.986834    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44\" (UID: \"c2eb9535-5242-4e53-af4a-3c03c5f56ff4\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"
Nov 01 06:44:44.987113 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:44.987014    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44\" (UID: \"c2eb9535-5242-4e53-af4a-3c03c5f56ff4\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"
Nov 01 06:44:44.987275 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:44.987250    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44\" (UID: \"c2eb9535-5242-4e53-af4a-3c03c5f56ff4\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"
Nov 01 06:44:45.002539 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.001419    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8f45a2\"/\"payload-processing-external-model-e2e--cb1d2723-dockercfg-bcrvz\""
Nov 01 06:44:45.005117 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.004908    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"]
Nov 01 06:44:45.088860 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.088824    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nm5r\" (UniqueName: \"kubernetes.io/projected/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-kube-api-access-2nm5r\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44\" (UID: \"c2eb9535-5242-4e53-af4a-3c03c5f56ff4\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"
Nov 01 06:44:45.088860 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.088910    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44\" (UID: \"c2eb9535-5242-4e53-af4a-3c03c5f56ff4\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"
Nov 01 06:44:45.088860 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.088964    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44\" (UID: \"c2eb9535-5242-4e53-af4a-3c03c5f56ff4\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"
Nov 01 06:44:45.088860 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:45.089023    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-47cacd68a67a: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:44:45.088860 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:45.089042    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:44:45.088860 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:45.089096    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-provider-credentials podName:c2eb9535-5242-4e53-af4a-3c03c5f56ff4 nodeName:}" failed. No retries permitted until 2026-11-01 06:44:45.589075157 +0000 UTC m=+3522.799898669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44" (UID: "c2eb9535-5242-4e53-af4a-3c03c5f56ff4") : secret "agc-cred-47cacd68a67a" not found
Nov 01 06:44:45.089644 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.089219    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44\" (UID: \"c2eb9535-5242-4e53-af4a-3c03c5f56ff4\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"
Nov 01 06:44:45.090108 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.090076    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44\" (UID: \"c2eb9535-5242-4e53-af4a-3c03c5f56ff4\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"
Nov 01 06:44:45.090460 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.090438    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44\" (UID: \"c2eb9535-5242-4e53-af4a-3c03c5f56ff4\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"
Nov 01 06:44:45.118596 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.118574    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"
Nov 01 06:44:45.121416 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.121392    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"
Nov 01 06:44:45.151748 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.151726    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nm5r\" (UniqueName: \"kubernetes.io/projected/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-kube-api-access-2nm5r\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44\" (UID: \"c2eb9535-5242-4e53-af4a-3c03c5f56ff4\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"
Nov 01 06:44:45.189688 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.189665    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7214a378-e37c-4625-b431-00f94c30799e-kube-api-access-rg4wf\" (UniqueName: \"kubernetes.io/projected/7214a378-e37c-4625-b431-00f94c30799e-kube-api-access-rg4wf\") pod \"7214a378-e37c-4625-b431-00f94c30799e\" (UID: \"7214a378-e37c-4625-b431-00f94c30799e\") "
Nov 01 06:44:45.189826 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.189701    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7214a378-e37c-4625-b431-00f94c30799e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7214a378-e37c-4625-b431-00f94c30799e-plugins-config-volume\") pod \"7214a378-e37c-4625-b431-00f94c30799e\" (UID: \"7214a378-e37c-4625-b431-00f94c30799e\") "
Nov 01 06:44:45.189826 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.189765    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-kube-api-access-xz4td\" (UniqueName: \"kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-kube-api-access-xz4td\") pod \"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1\" (UID: \"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1\") "
Nov 01 06:44:45.189935 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.189913    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7214a378-e37c-4625-b431-00f94c30799e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7214a378-e37c-4625-b431-00f94c30799e-provider-credentials\") pod \"7214a378-e37c-4625-b431-00f94c30799e\" (UID: \"7214a378-e37c-4625-b431-00f94c30799e\") "
Nov 01 06:44:45.189991 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.189954    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-routing-overlay-volume\") pod \"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1\" (UID: \"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1\") "
Nov 01 06:44:45.190288 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.190057    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-plugins-config-volume\") pod \"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1\" (UID: \"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1\") "
Nov 01 06:44:45.190288 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.190105    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-provider-credentials\") pod \"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1\" (UID: \"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1\") "
Nov 01 06:44:45.190288 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.190153    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7214a378-e37c-4625-b431-00f94c30799e-plugins-config-volume" pod "7214a378-e37c-4625-b431-00f94c30799e" (UID: "7214a378-e37c-4625-b431-00f94c30799e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:44:45.190288 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.190167    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7214a378-e37c-4625-b431-00f94c30799e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7214a378-e37c-4625-b431-00f94c30799e-routing-overlay-volume\") pod \"7214a378-e37c-4625-b431-00f94c30799e\" (UID: \"7214a378-e37c-4625-b431-00f94c30799e\") "
Nov 01 06:44:45.190596 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.190513    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7214a378-e37c-4625-b431-00f94c30799e-routing-overlay-volume" pod "7214a378-e37c-4625-b431-00f94c30799e" (UID: "7214a378-e37c-4625-b431-00f94c30799e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:44:45.190715 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.190694    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-routing-overlay-volume" pod "7afa361c-dc3c-4281-b43b-c7d4a7dfcac1" (UID: "7afa361c-dc3c-4281-b43b-c7d4a7dfcac1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:44:45.190887 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.190864    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-plugins-config-volume" pod "7afa361c-dc3c-4281-b43b-c7d4a7dfcac1" (UID: "7afa361c-dc3c-4281-b43b-c7d4a7dfcac1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:44:45.190958 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.190937    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7214a378-e37c-4625-b431-00f94c30799e-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:45.191018 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.190961    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7214a378-e37c-4625-b431-00f94c30799e-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:45.191018 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.190997    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:45.191976 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.191950    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-kube-api-access-xz4td" pod "7afa361c-dc3c-4281-b43b-c7d4a7dfcac1" (UID: "7afa361c-dc3c-4281-b43b-c7d4a7dfcac1"). InnerVolumeSpecName "kube-api-access-xz4td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:44:45.192118 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.192096    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7214a378-e37c-4625-b431-00f94c30799e-kube-api-access-rg4wf" pod "7214a378-e37c-4625-b431-00f94c30799e" (UID: "7214a378-e37c-4625-b431-00f94c30799e"). InnerVolumeSpecName "kube-api-access-rg4wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:44:45.192334 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.192303    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-provider-credentials" pod "7afa361c-dc3c-4281-b43b-c7d4a7dfcac1" (UID: "7afa361c-dc3c-4281-b43b-c7d4a7dfcac1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:44:45.192886 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.192871    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7214a378-e37c-4625-b431-00f94c30799e-provider-credentials" pod "7214a378-e37c-4625-b431-00f94c30799e" (UID: "7214a378-e37c-4625-b431-00f94c30799e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:44:45.221489 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.221464    5619 generic.go:356] "Generic (PLEG): container finished" podID="7214a378-e37c-4625-b431-00f94c30799e" containerID="450b15e2d4ee7cd09d7480f597862e447a3d4c345077ebd1d1fcd50f73686711" exitCode=0
Nov 01 06:44:45.221595 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.221534    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"
Nov 01 06:44:45.221595 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.221554    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl" event={"ID":"7214a378-e37c-4625-b431-00f94c30799e","Type":"ContainerDied","Data":"450b15e2d4ee7cd09d7480f597862e447a3d4c345077ebd1d1fcd50f73686711"}
Nov 01 06:44:45.221716 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.221614    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl" event={"ID":"7214a378-e37c-4625-b431-00f94c30799e","Type":"ContainerDied","Data":"c5b7081ffd5fdcbf74d671ba4b443df647557bb5d2c81840efb082212f8fd5fd"}
Nov 01 06:44:45.221716 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.221648    5619 scope.go:122] "RemoveContainer" containerID="450b15e2d4ee7cd09d7480f597862e447a3d4c345077ebd1d1fcd50f73686711"
Nov 01 06:44:45.222909 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.222889    5619 generic.go:356] "Generic (PLEG): container finished" podID="7afa361c-dc3c-4281-b43b-c7d4a7dfcac1" containerID="3c5fdc6903820c5a2db272f4dc1abede3138f5c423db3dc85481e0a5afa74a16" exitCode=0
Nov 01 06:44:45.223013 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.222951    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"
Nov 01 06:44:45.223077 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.223017    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9" event={"ID":"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1","Type":"ContainerDied","Data":"3c5fdc6903820c5a2db272f4dc1abede3138f5c423db3dc85481e0a5afa74a16"}
Nov 01 06:44:45.223077 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.223064    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9" event={"ID":"7afa361c-dc3c-4281-b43b-c7d4a7dfcac1","Type":"ContainerDied","Data":"d9b3c258f6d9263f0767fc76b959b6fcd83f7221b48dafda3aa07f4f4170103a"}
Nov 01 06:44:45.237886 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.237866    5619 scope.go:122] "RemoveContainer" containerID="450b15e2d4ee7cd09d7480f597862e447a3d4c345077ebd1d1fcd50f73686711"
Nov 01 06:44:45.238660 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:45.238193    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"450b15e2d4ee7cd09d7480f597862e447a3d4c345077ebd1d1fcd50f73686711\": container with ID starting with 450b15e2d4ee7cd09d7480f597862e447a3d4c345077ebd1d1fcd50f73686711 not found: ID does not exist" containerID="450b15e2d4ee7cd09d7480f597862e447a3d4c345077ebd1d1fcd50f73686711"
Nov 01 06:44:45.238660 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.238226    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450b15e2d4ee7cd09d7480f597862e447a3d4c345077ebd1d1fcd50f73686711"} err="failed to get container status \"450b15e2d4ee7cd09d7480f597862e447a3d4c345077ebd1d1fcd50f73686711\": rpc error: code = NotFound desc = could not find container \"450b15e2d4ee7cd09d7480f597862e447a3d4c345077ebd1d1fcd50f73686711\": container with ID starting with 450b15e2d4ee7cd09d7480f597862e447a3d4c345077ebd1d1fcd50f73686711 not found: ID does not exist"
Nov 01 06:44:45.238660 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.238248    5619 scope.go:122] "RemoveContainer" containerID="3c5fdc6903820c5a2db272f4dc1abede3138f5c423db3dc85481e0a5afa74a16"
Nov 01 06:44:45.249706 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.249667    5619 scope.go:122] "RemoveContainer" containerID="3c5fdc6903820c5a2db272f4dc1abede3138f5c423db3dc85481e0a5afa74a16"
Nov 01 06:44:45.250055 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:45.250030    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c5fdc6903820c5a2db272f4dc1abede3138f5c423db3dc85481e0a5afa74a16\": container with ID starting with 3c5fdc6903820c5a2db272f4dc1abede3138f5c423db3dc85481e0a5afa74a16 not found: ID does not exist" containerID="3c5fdc6903820c5a2db272f4dc1abede3138f5c423db3dc85481e0a5afa74a16"
Nov 01 06:44:45.250163 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.250063    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c5fdc6903820c5a2db272f4dc1abede3138f5c423db3dc85481e0a5afa74a16"} err="failed to get container status \"3c5fdc6903820c5a2db272f4dc1abede3138f5c423db3dc85481e0a5afa74a16\": rpc error: code = NotFound desc = could not find container \"3c5fdc6903820c5a2db272f4dc1abede3138f5c423db3dc85481e0a5afa74a16\": container with ID starting with 3c5fdc6903820c5a2db272f4dc1abede3138f5c423db3dc85481e0a5afa74a16 not found: ID does not exist"
Nov 01 06:44:45.291952 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.291923    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:45.291952 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.291952    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:45.292104 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.291968    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg4wf\" (UniqueName: \"kubernetes.io/projected/7214a378-e37c-4625-b431-00f94c30799e-kube-api-access-rg4wf\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:45.292104 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.291984    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz4td\" (UniqueName: \"kubernetes.io/projected/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1-kube-api-access-xz4td\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:45.292104 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.292000    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7214a378-e37c-4625-b431-00f94c30799e-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:45.416423 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.416384    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"]
Nov 01 06:44:45.422372 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.422337    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-fbbdnjpx9"]
Nov 01 06:44:45.533796 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.533760    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"]
Nov 01 06:44:45.540129 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.540101    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-5cb9c5nkl"]
Nov 01 06:44:45.596100 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:45.596071    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44\" (UID: \"c2eb9535-5242-4e53-af4a-3c03c5f56ff4\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"
Nov 01 06:44:45.596266 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:45.596192    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-47cacd68a67a: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:44:45.596266 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:45.596204    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:44:45.596266 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:45.596254    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-provider-credentials podName:c2eb9535-5242-4e53-af4a-3c03c5f56ff4 nodeName:}" failed. No retries permitted until 2026-11-01 06:44:46.596240198 +0000 UTC m=+3523.807063685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44" (UID: "c2eb9535-5242-4e53-af4a-3c03c5f56ff4") : secret "agc-cred-47cacd68a67a" not found
Nov 01 06:44:46.608623 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:46.608589    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44\" (UID: \"c2eb9535-5242-4e53-af4a-3c03c5f56ff4\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"
Nov 01 06:44:46.609025 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:46.608756    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-47cacd68a67a: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:44:46.609025 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:46.608775    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:44:46.609025 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:46.608832    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-provider-credentials podName:c2eb9535-5242-4e53-af4a-3c03c5f56ff4 nodeName:}" failed. No retries permitted until 2026-11-01 06:44:48.608816311 +0000 UTC m=+3525.819639816 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44" (UID: "c2eb9535-5242-4e53-af4a-3c03c5f56ff4") : secret "agc-cred-47cacd68a67a" not found
Nov 01 06:44:46.962279 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:46.962203    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7214a378-e37c-4625-b431-00f94c30799e" path="/var/lib/kubelet/pods/7214a378-e37c-4625-b431-00f94c30799e/volumes"
Nov 01 06:44:46.962636 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:46.962622    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7afa361c-dc3c-4281-b43b-c7d4a7dfcac1" path="/var/lib/kubelet/pods/7afa361c-dc3c-4281-b43b-c7d4a7dfcac1/volumes"
Nov 01 06:44:47.383664 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:47.383631    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"]
Nov 01 06:44:47.385725 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:47.383975    5619 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44" podUID="c2eb9535-5242-4e53-af4a-3c03c5f56ff4"
Nov 01 06:44:47.533146 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:47.533112    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:48.264901 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:48.261973    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"
Nov 01 06:44:48.273243 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:48.273207    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"
Nov 01 06:44:48.330014 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:48.329985    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-kube-api-access-2nm5r\" (UniqueName: \"kubernetes.io/projected/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-kube-api-access-2nm5r\") pod \"c2eb9535-5242-4e53-af4a-3c03c5f56ff4\" (UID: \"c2eb9535-5242-4e53-af4a-3c03c5f56ff4\") "
Nov 01 06:44:48.330169 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:48.330069    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-plugins-config-volume\") pod \"c2eb9535-5242-4e53-af4a-3c03c5f56ff4\" (UID: \"c2eb9535-5242-4e53-af4a-3c03c5f56ff4\") "
Nov 01 06:44:48.330270 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:48.330252    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-routing-overlay-volume\") pod \"c2eb9535-5242-4e53-af4a-3c03c5f56ff4\" (UID: \"c2eb9535-5242-4e53-af4a-3c03c5f56ff4\") "
Nov 01 06:44:48.330489 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:48.330450    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-plugins-config-volume" pod "c2eb9535-5242-4e53-af4a-3c03c5f56ff4" (UID: "c2eb9535-5242-4e53-af4a-3c03c5f56ff4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:44:48.330617 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:48.330576    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-routing-overlay-volume" pod "c2eb9535-5242-4e53-af4a-3c03c5f56ff4" (UID: "c2eb9535-5242-4e53-af4a-3c03c5f56ff4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:44:48.330759 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:48.330744    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:48.330799 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:48.330765    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:48.332133 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:48.332113    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-kube-api-access-2nm5r" pod "c2eb9535-5242-4e53-af4a-3c03c5f56ff4" (UID: "c2eb9535-5242-4e53-af4a-3c03c5f56ff4"). InnerVolumeSpecName "kube-api-access-2nm5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:44:48.432314 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:48.432283    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nm5r\" (UniqueName: \"kubernetes.io/projected/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-kube-api-access-2nm5r\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:48.560088 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:48.559996    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:48.635111 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:48.635066    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44\" (UID: \"c2eb9535-5242-4e53-af4a-3c03c5f56ff4\") " pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"
Nov 01 06:44:48.635317 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:48.635188    5619 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8f45a2/agc-cred-47cacd68a67a: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:44:48.635317 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:48.635210    5619 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44: secret "agc-cred-47cacd68a67a" not found
Nov 01 06:44:48.635317 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:48.635270    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-provider-credentials podName:c2eb9535-5242-4e53-af4a-3c03c5f56ff4 nodeName:}" failed. No retries permitted until 2026-11-01 06:44:52.635251193 +0000 UTC m=+3529.846074696 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44" (UID: "c2eb9535-5242-4e53-af4a-3c03c5f56ff4") : secret "agc-cred-47cacd68a67a" not found
Nov 01 06:44:49.267935 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:49.267852    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"
Nov 01 06:44:49.500084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:49.500049    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"]
Nov 01 06:44:49.507771 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:49.507744    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8f45a2/payload-processing-external-model-e2e-extmod-a-8f45a2-75f7zfl44"]
Nov 01 06:44:49.646169 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:49.646135    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2eb9535-5242-4e53-af4a-3c03c5f56ff4-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:44:50.037214 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:50.037126    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:50.184946 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:50.184907    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:50.963133 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:50.963104    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2eb9535-5242-4e53-af4a-3c03c5f56ff4" path="/var/lib/kubelet/pods/c2eb9535-5242-4e53-af4a-3c03c5f56ff4/volumes"
Nov 01 06:44:51.165813 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:51.165774    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:53.122668 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.122634    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"]
Nov 01 06:44:53.123233 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.123218    5619 cpu_manager.go:414] "containerMap: removing container" podUID="7214a378-e37c-4625-b431-00f94c30799e" containerName="payload-processing"
Nov 01 06:44:53.123277 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.123234    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7214a378-e37c-4625-b431-00f94c30799e" containerName="payload-processing"
Nov 01 06:44:53.123363 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.123349    5619 cpu_manager.go:414] "containerMap: removing container" podUID="7afa361c-dc3c-4281-b43b-c7d4a7dfcac1" containerName="payload-processing"
Nov 01 06:44:53.123406 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.123364    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7afa361c-dc3c-4281-b43b-c7d4a7dfcac1" containerName="payload-processing"
Nov 01 06:44:53.123496 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.123486    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="7afa361c-dc3c-4281-b43b-c7d4a7dfcac1" containerName="payload-processing"
Nov 01 06:44:53.123533 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.123499    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="7214a378-e37c-4625-b431-00f94c30799e" containerName="payload-processing"
Nov 01 06:44:53.126951 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.126935    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"
Nov 01 06:44:53.183383 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.183356    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-5be4fa-serving-cert\""
Nov 01 06:44:53.187127 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.187101    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"]
Nov 01 06:44:53.293773 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.293740    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfckb\" (UniqueName: \"kubernetes.io/projected/c9089f8a-a6b6-4c53-b483-8b5991330c33-kube-api-access-gfckb\") pod \"maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr\" (UID: \"c9089f8a-a6b6-4c53-b483-8b5991330c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"
Nov 01 06:44:53.293938 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.293792    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9089f8a-a6b6-4c53-b483-8b5991330c33-metrics-certs\") pod \"maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr\" (UID: \"c9089f8a-a6b6-4c53-b483-8b5991330c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"
Nov 01 06:44:53.293938 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.293888    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9089f8a-a6b6-4c53-b483-8b5991330c33-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr\" (UID: \"c9089f8a-a6b6-4c53-b483-8b5991330c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"
Nov 01 06:44:53.294047 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.294026    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9089f8a-a6b6-4c53-b483-8b5991330c33-maas-api-tls\") pod \"maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr\" (UID: \"c9089f8a-a6b6-4c53-b483-8b5991330c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"
Nov 01 06:44:53.309803 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.309770    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:53.394962 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.394866    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9089f8a-a6b6-4c53-b483-8b5991330c33-metrics-certs\") pod \"maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr\" (UID: \"c9089f8a-a6b6-4c53-b483-8b5991330c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"
Nov 01 06:44:53.394962 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.394945    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9089f8a-a6b6-4c53-b483-8b5991330c33-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr\" (UID: \"c9089f8a-a6b6-4c53-b483-8b5991330c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"
Nov 01 06:44:53.395202 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.395072    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9089f8a-a6b6-4c53-b483-8b5991330c33-maas-api-tls\") pod \"maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr\" (UID: \"c9089f8a-a6b6-4c53-b483-8b5991330c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"
Nov 01 06:44:53.395202 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.395175    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfckb\" (UniqueName: \"kubernetes.io/projected/c9089f8a-a6b6-4c53-b483-8b5991330c33-kube-api-access-gfckb\") pod \"maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr\" (UID: \"c9089f8a-a6b6-4c53-b483-8b5991330c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"
Nov 01 06:44:53.395446 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:53.395234    5619 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-serving-cert: secret "maas-api-e2e-extmod-a-5be4fa-serving-cert" not found
Nov 01 06:44:53.395446 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:53.395311    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9089f8a-a6b6-4c53-b483-8b5991330c33-maas-api-tls podName:c9089f8a-a6b6-4c53-b483-8b5991330c33 nodeName:}" failed. No retries permitted until 2026-11-01 06:44:53.895289288 +0000 UTC m=+3531.106112776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9089f8a-a6b6-4c53-b483-8b5991330c33-maas-api-tls") pod "maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr" (UID: "c9089f8a-a6b6-4c53-b483-8b5991330c33") : secret "maas-api-e2e-extmod-a-5be4fa-serving-cert" not found
Nov 01 06:44:53.395870 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.395843    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9089f8a-a6b6-4c53-b483-8b5991330c33-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr\" (UID: \"c9089f8a-a6b6-4c53-b483-8b5991330c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"
Nov 01 06:44:53.397868 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.397847    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9089f8a-a6b6-4c53-b483-8b5991330c33-metrics-certs\") pod \"maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr\" (UID: \"c9089f8a-a6b6-4c53-b483-8b5991330c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"
Nov 01 06:44:53.462354 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.459410    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfckb\" (UniqueName: \"kubernetes.io/projected/c9089f8a-a6b6-4c53-b483-8b5991330c33-kube-api-access-gfckb\") pod \"maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr\" (UID: \"c9089f8a-a6b6-4c53-b483-8b5991330c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"
Nov 01 06:44:53.465121 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.465088    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:53.643231 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.643194    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:53.901944 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.901913    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9089f8a-a6b6-4c53-b483-8b5991330c33-maas-api-tls\") pod \"maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr\" (UID: \"c9089f8a-a6b6-4c53-b483-8b5991330c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"
Nov 01 06:44:53.904418 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:53.904398    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9089f8a-a6b6-4c53-b483-8b5991330c33-maas-api-tls\") pod \"maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr\" (UID: \"c9089f8a-a6b6-4c53-b483-8b5991330c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"
Nov 01 06:44:54.039511 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:54.039481    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"
Nov 01 06:44:54.418816 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:54.418790    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"]
Nov 01 06:44:54.421906 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:44:54.421874    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9089f8a_a6b6_4c53_b483_8b5991330c33.slice/crio-9c28dd433ffbc85fb6cb8a455833531b836157038149f7683d423b6deb1e17ca WatchSource:0}: Error finding container 9c28dd433ffbc85fb6cb8a455833531b836157038149f7683d423b6deb1e17ca: Status 404 returned error can't find the container with id 9c28dd433ffbc85fb6cb8a455833531b836157038149f7683d423b6deb1e17ca
Nov 01 06:44:54.423279 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:54.423258    5619 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 06:44:55.303242 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:55.303208    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr" event={"ID":"c9089f8a-a6b6-4c53-b483-8b5991330c33","Type":"ContainerStarted","Data":"9c28dd433ffbc85fb6cb8a455833531b836157038149f7683d423b6deb1e17ca"}
Nov 01 06:44:55.879792 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:44:55.879748    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:44:55.918918 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:55.918882    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:56.059090 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:56.059051    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:44:56.311994 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:56.311915    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr" event={"ID":"c9089f8a-a6b6-4c53-b483-8b5991330c33","Type":"ContainerStarted","Data":"cdd9cd9b9dce913f7b01cf4381146db3f641f8cd556ce25a40f8a6f05247189b"}
Nov 01 06:44:56.312145 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:56.312121    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"
Nov 01 06:44:56.459846 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:56.459780    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr" podStartSLOduration=2.509847706 podStartE2EDuration="3.459764252s" totalImagesPullingTime="949.916546ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:44:56.45924497 +0000 UTC m=+3533.670068481" watchObservedRunningTime="2026-11-01 06:44:56.459764252 +0000 UTC m=+3533.670587762"
Nov 01 06:44:58.252644 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:44:58.252609    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:45:02.318466 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:02.318429    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr" podUID="c9089f8a-a6b6-4c53-b483-8b5991330c33" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.151:8443/health\": dial tcp 10.133.0.151:8443: connect: connection refused"
Nov 01 06:45:02.966724 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:02.966688    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:45:06.375122 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:06.375087    5619 generic.go:356] "Generic (PLEG): container finished" podID="c9089f8a-a6b6-4c53-b483-8b5991330c33" containerID="cdd9cd9b9dce913f7b01cf4381146db3f641f8cd556ce25a40f8a6f05247189b" exitCode=1
Nov 01 06:45:06.375578 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:06.375157    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr" event={"ID":"c9089f8a-a6b6-4c53-b483-8b5991330c33","Type":"ContainerDied","Data":"cdd9cd9b9dce913f7b01cf4381146db3f641f8cd556ce25a40f8a6f05247189b"}
Nov 01 06:45:06.375635 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:06.375617    5619 scope.go:122] "RemoveContainer" containerID="cdd9cd9b9dce913f7b01cf4381146db3f641f8cd556ce25a40f8a6f05247189b"
Nov 01 06:45:07.383567 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:07.383519    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr" event={"ID":"c9089f8a-a6b6-4c53-b483-8b5991330c33","Type":"ContainerStarted","Data":"c32ff1c10c1842bd376508025682375bcd98123d2812e0bfaacedfc35f90583b"}
Nov 01 06:45:07.383956 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:07.383762    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"
Nov 01 06:45:07.758982 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:07.758945    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:45:12.460919 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:12.460884    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:45:13.397194 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:13.397157    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"
Nov 01 06:45:17.180343 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:17.180285    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:45:21.896315 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:21.896271    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:45:25.879813 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:45:25.879764    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:45:26.626556 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:26.626521    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:45:31.312626 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:31.312584    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:45:32.306636 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:32.306595    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:45:35.268215 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:35.268176    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:45:36.049779 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:36.049737    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:45:36.710024 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:36.709984    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:45:40.773582 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:40.773545    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:45:42.503605 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.503573    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"]
Nov 01 06:45:42.507487 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.507470    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:42.560924 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.560893    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-14a6e4\"/\"e2e-conflict-a-be936817-kserve-self-signed-certs\""
Nov 01 06:45:42.565435 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.565408    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-home\") pod \"e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:42.565663 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.565641    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-model-cache\") pod \"e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:42.565848 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.565831    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5532f87-20d8-4fdf-89ce-28ed181176a4-tls-certs\") pod \"e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:42.566098 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.566032    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-tmp-dir\") pod \"e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:42.566219 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.566133    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9vn6\" (UniqueName: \"kubernetes.io/projected/d5532f87-20d8-4fdf-89ce-28ed181176a4-kube-api-access-l9vn6\") pod \"e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:42.566219 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.566169    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-dshm\") pod \"e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:42.602084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.602048    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:45:42.611792 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.611759    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-14a6e4\"/\"openshift-service-ca.crt\""
Nov 01 06:45:42.611792 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.611785    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-14a6e4\"/\"kube-root-ca.crt\""
Nov 01 06:45:42.614258 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.614239    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"]
Nov 01 06:45:42.662011 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.661980    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-14a6e4\"/\"default-dockercfg-qptrp\""
Nov 01 06:45:42.669144 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.667914    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-dshm\") pod \"e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:42.669144 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.668030    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-home\") pod \"e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:42.669144 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.668073    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-model-cache\") pod \"e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:42.669144 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.668151    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5532f87-20d8-4fdf-89ce-28ed181176a4-tls-certs\") pod \"e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:42.669144 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.668296    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-tmp-dir\") pod \"e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:42.669144 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.668412    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9vn6\" (UniqueName: \"kubernetes.io/projected/d5532f87-20d8-4fdf-89ce-28ed181176a4-kube-api-access-l9vn6\") pod \"e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:42.669751 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.669197    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-tmp-dir\") pod \"e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:42.669751 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.669448    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-home\") pod \"e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:42.669751 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.669531    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-model-cache\") pod \"e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:42.671104 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.671077    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-dshm\") pod \"e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:42.676836 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.676808    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5532f87-20d8-4fdf-89ce-28ed181176a4-tls-certs\") pod \"e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:42.730124 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.730093    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9vn6\" (UniqueName: \"kubernetes.io/projected/d5532f87-20d8-4fdf-89ce-28ed181176a4-kube-api-access-l9vn6\") pod \"e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:42.818944 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:42.818864    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:43.195901 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:43.195875    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"]
Nov 01 06:45:43.196719 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:45:43.196689    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5532f87_20d8_4fdf_89ce_28ed181176a4.slice/crio-2ceef1fe17e9a33a65f58f5129e62cc5ada7ceea6fdead3c3f873df5005aba04 WatchSource:0}: Error finding container 2ceef1fe17e9a33a65f58f5129e62cc5ada7ceea6fdead3c3f873df5005aba04: Status 404 returned error can't find the container with id 2ceef1fe17e9a33a65f58f5129e62cc5ada7ceea6fdead3c3f873df5005aba04
Nov 01 06:45:43.606785 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:43.606752    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8" event={"ID":"d5532f87-20d8-4fdf-89ce-28ed181176a4","Type":"ContainerStarted","Data":"dd69b0951ff227a19c020cd4063fff1ba8110ca27f4e96a63431d6202e829510"}
Nov 01 06:45:43.607263 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:43.606800    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8" event={"ID":"d5532f87-20d8-4fdf-89ce-28ed181176a4","Type":"ContainerStarted","Data":"2ceef1fe17e9a33a65f58f5129e62cc5ada7ceea6fdead3c3f873df5005aba04"}
Nov 01 06:45:43.725516 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:43.725461    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8" podStartSLOduration=1.725446658 podStartE2EDuration="1.725446658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:43.724435898 +0000 UTC m=+3580.935259407" watchObservedRunningTime="2026-11-01 06:45:43.725446658 +0000 UTC m=+3580.936270186"
Nov 01 06:45:43.819424 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:43.819381    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:45.487245 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:45.487204    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:45:47.290709 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:47.290676    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:45:50.276503 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:50.276458    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:45:51.481870 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:51.481830    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:45:51.676000 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:51.675962    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:45:52.249039 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:52.249009    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"]
Nov 01 06:45:52.273856 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:52.273825    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"]
Nov 01 06:45:52.413519 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:52.413492    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:45:52.819000 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:52.818966    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:52.831498 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:52.831474    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:53.688290 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:53.688258    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:45:55.080041 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:55.080009    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:45:55.880572 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:45:55.880536    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:45:58.533946 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:58.533906    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:45:59.992020 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:45:59.991975    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:00.841980 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:00.841941    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:02.055555 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:02.055518    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:02.762692 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:02.762651    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:02.916167 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:02.916129    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:05.057778 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:05.057746    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:05.222239 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:05.222202    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:05.833113 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:05.833063    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:05.979292 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:05.979254    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:07.269902 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.269867    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8" podUID="fb255dfb-b714-4f2e-98c2-e8aa8695b9da" containerName="main" containerID="cri-o://e7e55af8de06f327898327f3140311064a0136561ccb971644c0231dbe87b1da" gracePeriod=45
Nov 01 06:46:07.293370 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.293338    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2" podUID="8e8d332c-35b4-40b7-bf20-7623f1bc19d9" containerName="main" containerID="cri-o://9f2a97c51a512caab00022ca8643390cc7557ed25aded6fd9c10a5ee5886777e" gracePeriod=45
Nov 01 06:46:07.378830 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.378801    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2" podUID="8e8d332c-35b4-40b7-bf20-7623f1bc19d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": dial tcp 10.133.0.42:8000: connect: connection refused"
Nov 01 06:46:07.396218 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.396189    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8" podUID="fb255dfb-b714-4f2e-98c2-e8aa8695b9da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": dial tcp 10.133.0.50:8000: connect: connection refused"
Nov 01 06:46:07.558681 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.558656    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:46:07.587406 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.587385    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 06:46:07.655518 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.655490    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-tls-certs\") pod \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") "
Nov 01 06:46:07.655645 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.655526    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-model-cache\") pod \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") "
Nov 01 06:46:07.655645 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.655601    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-kube-api-access-grh6s\" (UniqueName: \"kubernetes.io/projected/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-kube-api-access-grh6s\") pod \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") "
Nov 01 06:46:07.655645 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.655632    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-home\" (UniqueName: \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-home\") pod \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") "
Nov 01 06:46:07.655790 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.655667    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-home\" (UniqueName: \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-home\") pod \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") "
Nov 01 06:46:07.655790 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.655695    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-dshm\") pod \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") "
Nov 01 06:46:07.655790 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.655716    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-model-cache\") pod \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") "
Nov 01 06:46:07.655790 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.655750    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-tmp-dir\") pod \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") "
Nov 01 06:46:07.656012 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.655789    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-tmp-dir\") pod \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") "
Nov 01 06:46:07.656012 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.655833    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-kube-api-access-d9zbk\" (UniqueName: \"kubernetes.io/projected/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-kube-api-access-d9zbk\") pod \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") "
Nov 01 06:46:07.656012 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.655875    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-dshm\") pod \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\" (UID: \"fb255dfb-b714-4f2e-98c2-e8aa8695b9da\") "
Nov 01 06:46:07.656012 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.655965    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-home" pod "8e8d332c-35b4-40b7-bf20-7623f1bc19d9" (UID: "8e8d332c-35b4-40b7-bf20-7623f1bc19d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:07.656012 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.655979    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-home" pod "fb255dfb-b714-4f2e-98c2-e8aa8695b9da" (UID: "fb255dfb-b714-4f2e-98c2-e8aa8695b9da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:07.656012 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.656002    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-tls-certs\") pod \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\" (UID: \"8e8d332c-35b4-40b7-bf20-7623f1bc19d9\") "
Nov 01 06:46:07.656308 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.656022    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-model-cache" pod "fb255dfb-b714-4f2e-98c2-e8aa8695b9da" (UID: "fb255dfb-b714-4f2e-98c2-e8aa8695b9da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:07.656308 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.656169    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-tmp-dir" pod "fb255dfb-b714-4f2e-98c2-e8aa8695b9da" (UID: "fb255dfb-b714-4f2e-98c2-e8aa8695b9da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:07.656308 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.656176    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-tmp-dir" pod "8e8d332c-35b4-40b7-bf20-7623f1bc19d9" (UID: "8e8d332c-35b4-40b7-bf20-7623f1bc19d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:07.656465 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.656373    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-model-cache" pod "8e8d332c-35b4-40b7-bf20-7623f1bc19d9" (UID: "8e8d332c-35b4-40b7-bf20-7623f1bc19d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:07.656620 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.656600    5619 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-model-cache\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.656722 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.656626    5619 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-home\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.656722 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.656642    5619 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-home\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.656722 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.656654    5619 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-model-cache\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.656722 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.656668    5619 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-tmp-dir\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.656722 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.656682    5619 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-tmp-dir\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.658301 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.658273    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-tls-certs" pod "8e8d332c-35b4-40b7-bf20-7623f1bc19d9" (UID: "8e8d332c-35b4-40b7-bf20-7623f1bc19d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:46:07.658431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.658393    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-dshm" pod "8e8d332c-35b4-40b7-bf20-7623f1bc19d9" (UID: "8e8d332c-35b4-40b7-bf20-7623f1bc19d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:07.658431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.658404    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-kube-api-access-d9zbk" pod "fb255dfb-b714-4f2e-98c2-e8aa8695b9da" (UID: "fb255dfb-b714-4f2e-98c2-e8aa8695b9da"). InnerVolumeSpecName "kube-api-access-d9zbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:46:07.658431 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.658414    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-tls-certs" pod "fb255dfb-b714-4f2e-98c2-e8aa8695b9da" (UID: "fb255dfb-b714-4f2e-98c2-e8aa8695b9da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:46:07.658541 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.658479    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-kube-api-access-grh6s" pod "8e8d332c-35b4-40b7-bf20-7623f1bc19d9" (UID: "8e8d332c-35b4-40b7-bf20-7623f1bc19d9"). InnerVolumeSpecName "kube-api-access-grh6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:46:07.658625 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.658605    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-dshm" pod "fb255dfb-b714-4f2e-98c2-e8aa8695b9da" (UID: "fb255dfb-b714-4f2e-98c2-e8aa8695b9da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:07.760430 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.760389    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grh6s\" (UniqueName: \"kubernetes.io/projected/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-kube-api-access-grh6s\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.760430 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.760429    5619 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-dshm\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.760683 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.760446    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9zbk\" (UniqueName: \"kubernetes.io/projected/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-kube-api-access-d9zbk\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.760683 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.760460    5619 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-dshm\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.760683 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.760474    5619 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8d332c-35b4-40b7-bf20-7623f1bc19d9-tls-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.760683 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.760489    5619 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb255dfb-b714-4f2e-98c2-e8aa8695b9da-tls-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:07.766047 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.766016    5619 generic.go:356] "Generic (PLEG): container finished" podID="8e8d332c-35b4-40b7-bf20-7623f1bc19d9" containerID="9f2a97c51a512caab00022ca8643390cc7557ed25aded6fd9c10a5ee5886777e" exitCode=0
Nov 01 06:46:07.766208 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.766074    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"
Nov 01 06:46:07.766208 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.766101    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2" event={"ID":"8e8d332c-35b4-40b7-bf20-7623f1bc19d9","Type":"ContainerDied","Data":"9f2a97c51a512caab00022ca8643390cc7557ed25aded6fd9c10a5ee5886777e"}
Nov 01 06:46:07.766208 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.766160    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2" event={"ID":"8e8d332c-35b4-40b7-bf20-7623f1bc19d9","Type":"ContainerDied","Data":"c9202bca7bf2a3ae65153a07000d5a512a5b350381b3f1ae8763f367cd25ba5f"}
Nov 01 06:46:07.766208 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.766184    5619 scope.go:122] "RemoveContainer" containerID="9f2a97c51a512caab00022ca8643390cc7557ed25aded6fd9c10a5ee5886777e"
Nov 01 06:46:07.768063 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.768035    5619 generic.go:356] "Generic (PLEG): container finished" podID="fb255dfb-b714-4f2e-98c2-e8aa8695b9da" containerID="e7e55af8de06f327898327f3140311064a0136561ccb971644c0231dbe87b1da" exitCode=0
Nov 01 06:46:07.768208 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.768081    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8" event={"ID":"fb255dfb-b714-4f2e-98c2-e8aa8695b9da","Type":"ContainerDied","Data":"e7e55af8de06f327898327f3140311064a0136561ccb971644c0231dbe87b1da"}
Nov 01 06:46:07.768208 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.768110    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8" event={"ID":"fb255dfb-b714-4f2e-98c2-e8aa8695b9da","Type":"ContainerDied","Data":"84b0d6f7de2ba89430125dc9ea7c1d02bdbaeefa1aa77447a1b87920241b7c42"}
Nov 01 06:46:07.768208 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.768157    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"
Nov 01 06:46:07.779400 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.779382    5619 scope.go:122] "RemoveContainer" containerID="9f2a97c51a512caab00022ca8643390cc7557ed25aded6fd9c10a5ee5886777e"
Nov 01 06:46:07.779661 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:46:07.779642    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f2a97c51a512caab00022ca8643390cc7557ed25aded6fd9c10a5ee5886777e\": container with ID starting with 9f2a97c51a512caab00022ca8643390cc7557ed25aded6fd9c10a5ee5886777e not found: ID does not exist" containerID="9f2a97c51a512caab00022ca8643390cc7557ed25aded6fd9c10a5ee5886777e"
Nov 01 06:46:07.779754 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.779673    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f2a97c51a512caab00022ca8643390cc7557ed25aded6fd9c10a5ee5886777e"} err="failed to get container status \"9f2a97c51a512caab00022ca8643390cc7557ed25aded6fd9c10a5ee5886777e\": rpc error: code = NotFound desc = could not find container \"9f2a97c51a512caab00022ca8643390cc7557ed25aded6fd9c10a5ee5886777e\": container with ID starting with 9f2a97c51a512caab00022ca8643390cc7557ed25aded6fd9c10a5ee5886777e not found: ID does not exist"
Nov 01 06:46:07.779754 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.779696    5619 scope.go:122] "RemoveContainer" containerID="e7e55af8de06f327898327f3140311064a0136561ccb971644c0231dbe87b1da"
Nov 01 06:46:07.789306 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.789290    5619 scope.go:122] "RemoveContainer" containerID="e7e55af8de06f327898327f3140311064a0136561ccb971644c0231dbe87b1da"
Nov 01 06:46:07.789602 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:46:07.789574    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7e55af8de06f327898327f3140311064a0136561ccb971644c0231dbe87b1da\": container with ID starting with e7e55af8de06f327898327f3140311064a0136561ccb971644c0231dbe87b1da not found: ID does not exist" containerID="e7e55af8de06f327898327f3140311064a0136561ccb971644c0231dbe87b1da"
Nov 01 06:46:07.789734 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.789603    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7e55af8de06f327898327f3140311064a0136561ccb971644c0231dbe87b1da"} err="failed to get container status \"e7e55af8de06f327898327f3140311064a0136561ccb971644c0231dbe87b1da\": rpc error: code = NotFound desc = could not find container \"e7e55af8de06f327898327f3140311064a0136561ccb971644c0231dbe87b1da\": container with ID starting with e7e55af8de06f327898327f3140311064a0136561ccb971644c0231dbe87b1da not found: ID does not exist"
Nov 01 06:46:07.908920 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.908883    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"]
Nov 01 06:46:07.913116 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:07.913090    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/simulator-w4-f73e89-kserve-775bd847dc-qz5h2"]
Nov 01 06:46:08.023746 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:08.023715    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"]
Nov 01 06:46:08.028122 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:08.028096    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f73e89/distinct-2-w4-f73e89-kserve-bd6475978-nkqm8"]
Nov 01 06:46:08.962958 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:08.962925    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e8d332c-35b4-40b7-bf20-7623f1bc19d9" path="/var/lib/kubelet/pods/8e8d332c-35b4-40b7-bf20-7623f1bc19d9/volumes"
Nov 01 06:46:08.963389 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:08.963373    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb255dfb-b714-4f2e-98c2-e8aa8695b9da" path="/var/lib/kubelet/pods/fb255dfb-b714-4f2e-98c2-e8aa8695b9da/volumes"
Nov 01 06:46:10.138165 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:10.138122    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:11.159079 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.158992    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:11.229140 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.229106    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"]
Nov 01 06:46:11.229705 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.229679    5619 cpu_manager.go:414] "containerMap: removing container" podUID="fb255dfb-b714-4f2e-98c2-e8aa8695b9da" containerName="main"
Nov 01 06:46:11.229705 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.229696    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb255dfb-b714-4f2e-98c2-e8aa8695b9da" containerName="main"
Nov 01 06:46:11.229705 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.229709    5619 cpu_manager.go:414] "containerMap: removing container" podUID="8e8d332c-35b4-40b7-bf20-7623f1bc19d9" containerName="main"
Nov 01 06:46:11.229954 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.229715    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e8d332c-35b4-40b7-bf20-7623f1bc19d9" containerName="main"
Nov 01 06:46:11.229954 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.229824    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb255dfb-b714-4f2e-98c2-e8aa8695b9da" containerName="main"
Nov 01 06:46:11.229954 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.229832    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e8d332c-35b4-40b7-bf20-7623f1bc19d9" containerName="main"
Nov 01 06:46:11.234513 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.234496    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:11.299281 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.299245    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wppj9\" (UniqueName: \"kubernetes.io/projected/c2a12b29-2d63-40e9-8704-099d289a05ec-kube-api-access-wppj9\") pod \"e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:11.299495 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.299408    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2a12b29-2d63-40e9-8704-099d289a05ec-tls-certs\") pod \"e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:11.299576 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.299508    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-model-cache\") pod \"e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:11.299576 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.299558    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-dshm\") pod \"e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:11.299699 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.299609    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-tmp-dir\") pod \"e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:11.299699 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.299659    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-home\") pod \"e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:11.339441 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.339409    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-14a6e4\"/\"e2e-conflict-b-be936817-kserve-self-signed-certs\""
Nov 01 06:46:11.341992 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.341969    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"]
Nov 01 06:46:11.354023 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.353993    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:11.404347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.401563    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-model-cache\") pod \"e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:11.404347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.402016    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-model-cache\") pod \"e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:11.404347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.402102    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-dshm\") pod \"e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:11.404347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.402716    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-tmp-dir\") pod \"e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:11.404347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.403475    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-tmp-dir\") pod \"e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:11.404347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.403581    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-home\") pod \"e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:11.404347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.403753    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wppj9\" (UniqueName: \"kubernetes.io/projected/c2a12b29-2d63-40e9-8704-099d289a05ec-kube-api-access-wppj9\") pod \"e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:11.404347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.403831    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2a12b29-2d63-40e9-8704-099d289a05ec-tls-certs\") pod \"e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:11.404935 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.404562    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-home\") pod \"e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:11.406075 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.405080    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-dshm\") pod \"e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:11.410593 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.410526    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2a12b29-2d63-40e9-8704-099d289a05ec-tls-certs\") pod \"e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:11.468955 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.468926    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wppj9\" (UniqueName: \"kubernetes.io/projected/c2a12b29-2d63-40e9-8704-099d289a05ec-kube-api-access-wppj9\") pod \"e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") " pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:11.547202 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.547169    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:11.956432 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:11.956400    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"]
Nov 01 06:46:11.957806 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:46:11.957774    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2a12b29_2d63_40e9_8704_099d289a05ec.slice/crio-79ded5acfa06658912ca25db50aa2000d22c08faff566348c68256bafb9fdfea WatchSource:0}: Error finding container 79ded5acfa06658912ca25db50aa2000d22c08faff566348c68256bafb9fdfea: Status 404 returned error can't find the container with id 79ded5acfa06658912ca25db50aa2000d22c08faff566348c68256bafb9fdfea
Nov 01 06:46:12.804969 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:12.804932    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t" event={"ID":"c2a12b29-2d63-40e9-8704-099d289a05ec","Type":"ContainerStarted","Data":"a26d0ed3fe5c4c5dfee1d5cf5fa4c7570054ed7bbeef569c4d65612b54005904"}
Nov 01 06:46:12.805472 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:12.804978    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t" event={"ID":"c2a12b29-2d63-40e9-8704-099d289a05ec","Type":"ContainerStarted","Data":"79ded5acfa06658912ca25db50aa2000d22c08faff566348c68256bafb9fdfea"}
Nov 01 06:46:12.924884 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:12.924829    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t" podStartSLOduration=1.924811646 podStartE2EDuration="1.924811646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:46:12.922522111 +0000 UTC m=+3610.133345613" watchObservedRunningTime="2026-11-01 06:46:12.924811646 +0000 UTC m=+3610.135635155"
Nov 01 06:46:13.547798 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:13.547766    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:14.885521 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:14.885486    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:17.015966 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:17.015929    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:17.556853 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:17.556782    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:17.732806 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:17.732771    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:17.879696 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:17.879656    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:20.013790 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:20.013735    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:20.735800 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:20.735755    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:20.910972 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:20.910933    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:21.547349 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:21.547280    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:21.560678 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:21.560655    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:21.881717 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:21.881686    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:46:23.012111 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.012076    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"]
Nov 01 06:46:23.019110 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.019085    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.124930 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.124892    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-14a6e4-openshift-default-dockercfg-vxhc4\""
Nov 01 06:46:23.128770 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.128738    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"]
Nov 01 06:46:23.148605 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.148578    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.148727 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.148613    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.148727 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.148646    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.148804 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.148785    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.148904 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.148888    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96534e12-7487-48ec-bfaa-9d7cc14181d2-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.148944 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.148915    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.149022 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.148989    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zlqz\" (UniqueName: \"kubernetes.io/projected/96534e12-7487-48ec-bfaa-9d7cc14181d2-kube-api-access-5zlqz\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.149022 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.149015    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.149092 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.149076    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.249815 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.249785    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96534e12-7487-48ec-bfaa-9d7cc14181d2-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.249990 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.249821    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.249990 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.249849    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zlqz\" (UniqueName: \"kubernetes.io/projected/96534e12-7487-48ec-bfaa-9d7cc14181d2-kube-api-access-5zlqz\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.249990 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.249879    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.249990 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.249917    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.250286 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.250011    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.250286 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.250141    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.250286 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.250223    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.250495 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.250291    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.250495 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.250358    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-credential-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.250495 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.250477    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-data\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.250667 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.250523    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-workload-socket\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.250667 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.250650    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-workload-certs\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.250749 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.250721    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96534e12-7487-48ec-bfaa-9d7cc14181d2-istiod-ca-cert\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.252210 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.252190    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-envoy\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.252615 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.252596    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-podinfo\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.311597 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.311535    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-token\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.311875 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.311851    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zlqz\" (UniqueName: \"kubernetes.io/projected/96534e12-7487-48ec-bfaa-9d7cc14181d2-kube-api-access-5zlqz\") pod \"e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") " pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.333891 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.333868    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:23.765133 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.765098    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"]
Nov 01 06:46:23.799726 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:46:23.799698    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96534e12_7487_48ec_bfaa_9d7cc14181d2.slice/crio-0d821f9a54dd0eb9ff9ddac5f39015697423efa46abe255ac77f08053ffe647d WatchSource:0}: Error finding container 0d821f9a54dd0eb9ff9ddac5f39015697423efa46abe255ac77f08053ffe647d: Status 404 returned error can't find the container with id 0d821f9a54dd0eb9ff9ddac5f39015697423efa46abe255ac77f08053ffe647d
Nov 01 06:46:23.802124 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.802086    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:46:23.802217 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.802162    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:46:23.802217 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.802194    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:46:23.889304 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.889269    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f" event={"ID":"96534e12-7487-48ec-bfaa-9d7cc14181d2","Type":"ContainerStarted","Data":"7ce9adb6453fa7caa91f215697ef08e0206fb6bf445e170e723b09fddc07b4f9"}
Nov 01 06:46:23.889444 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:23.889379    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f" event={"ID":"96534e12-7487-48ec-bfaa-9d7cc14181d2","Type":"ContainerStarted","Data":"0d821f9a54dd0eb9ff9ddac5f39015697423efa46abe255ac77f08053ffe647d"}
Nov 01 06:46:24.012573 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:24.012512    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f" podStartSLOduration=2.012496997 podStartE2EDuration="2.012496997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:46:24.010698024 +0000 UTC m=+3621.221521530" watchObservedRunningTime="2026-11-01 06:46:24.012496997 +0000 UTC m=+3621.223320563"
Nov 01 06:46:24.334735 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:24.334691    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:24.336199 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:24.336168    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 01 06:46:24.336463 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:24.336433    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f" podUID="96534e12-7487-48ec-bfaa-9d7cc14181d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 01 06:46:25.136179 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:25.136137    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:25.334737 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:25.334705    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 01 06:46:25.334911 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:25.334769    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f" podUID="96534e12-7487-48ec-bfaa-9d7cc14181d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 01 06:46:25.903700 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:46:25.903660    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:46:26.334888 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:26.334804    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 01 06:46:26.334888 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:26.334867    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f" podUID="96534e12-7487-48ec-bfaa-9d7cc14181d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 01 06:46:27.334800 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:27.334765    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 01 06:46:27.334954 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:27.334818    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f" podUID="96534e12-7487-48ec-bfaa-9d7cc14181d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 01 06:46:28.334582 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:28.334550    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 01 06:46:28.334773 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:28.334602    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f" podUID="96534e12-7487-48ec-bfaa-9d7cc14181d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 01 06:46:29.140679 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.140643    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:29.198036 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.198003    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"]
Nov 01 06:46:29.198285 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.198258    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m" podUID="d7e166f7-3bf1-4d53-b9e8-e964a5546a8d" containerName="maas-api" containerID="cri-o://aacf1f51772e84e5b2b0353a1521470408a11a830a540c00765118de582a1fcc" gracePeriod=30
Nov 01 06:46:29.297010 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.296974    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:29.335548 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.335058    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 01 06:46:29.335548 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.335138    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f" podUID="96534e12-7487-48ec-bfaa-9d7cc14181d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 01 06:46:29.505589 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.505531    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:29.613826 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.613531    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"
Nov 01 06:46:29.687537 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.687446    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-maas-api-tls\") pod \"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d\" (UID: \"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d\") "
Nov 01 06:46:29.687537 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.687527    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-openshift-service-ca\") pod \"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d\" (UID: \"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d\") "
Nov 01 06:46:29.687763 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.687650    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-kube-api-access-j5c9g\" (UniqueName: \"kubernetes.io/projected/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-kube-api-access-j5c9g\") pod \"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d\" (UID: \"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d\") "
Nov 01 06:46:29.687763 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.687709    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-metrics-certs\") pod \"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d\" (UID: \"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d\") "
Nov 01 06:46:29.688531 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.688487    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-openshift-service-ca" pod "d7e166f7-3bf1-4d53-b9e8-e964a5546a8d" (UID: "d7e166f7-3bf1-4d53-b9e8-e964a5546a8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:46:29.690228 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.690198    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-metrics-certs" pod "d7e166f7-3bf1-4d53-b9e8-e964a5546a8d" (UID: "d7e166f7-3bf1-4d53-b9e8-e964a5546a8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:46:29.690538 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.690516    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-maas-api-tls" pod "d7e166f7-3bf1-4d53-b9e8-e964a5546a8d" (UID: "d7e166f7-3bf1-4d53-b9e8-e964a5546a8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:46:29.690716 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.690690    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-kube-api-access-j5c9g" pod "d7e166f7-3bf1-4d53-b9e8-e964a5546a8d" (UID: "d7e166f7-3bf1-4d53-b9e8-e964a5546a8d"). InnerVolumeSpecName "kube-api-access-j5c9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:46:29.789842 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.789795    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:29.789842 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.789841    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5c9g\" (UniqueName: \"kubernetes.io/projected/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-kube-api-access-j5c9g\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:29.790095 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.789857    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:29.790095 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.789870    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:29.930117 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.930080    5619 generic.go:356] "Generic (PLEG): container finished" podID="d7e166f7-3bf1-4d53-b9e8-e964a5546a8d" containerID="aacf1f51772e84e5b2b0353a1521470408a11a830a540c00765118de582a1fcc" exitCode=0
Nov 01 06:46:29.930278 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.930138    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"
Nov 01 06:46:29.930278 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.930161    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m" event={"ID":"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d","Type":"ContainerDied","Data":"aacf1f51772e84e5b2b0353a1521470408a11a830a540c00765118de582a1fcc"}
Nov 01 06:46:29.930278 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.930213    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m" event={"ID":"d7e166f7-3bf1-4d53-b9e8-e964a5546a8d","Type":"ContainerDied","Data":"da3b54e14110d484284025f9bb05c10cdcc7a7cf95adb379996254063f76ec87"}
Nov 01 06:46:29.930278 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.930234    5619 scope.go:122] "RemoveContainer" containerID="aacf1f51772e84e5b2b0353a1521470408a11a830a540c00765118de582a1fcc"
Nov 01 06:46:29.941094 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.941077    5619 scope.go:122] "RemoveContainer" containerID="aacf1f51772e84e5b2b0353a1521470408a11a830a540c00765118de582a1fcc"
Nov 01 06:46:29.941396 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:46:29.941371    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aacf1f51772e84e5b2b0353a1521470408a11a830a540c00765118de582a1fcc\": container with ID starting with aacf1f51772e84e5b2b0353a1521470408a11a830a540c00765118de582a1fcc not found: ID does not exist" containerID="aacf1f51772e84e5b2b0353a1521470408a11a830a540c00765118de582a1fcc"
Nov 01 06:46:29.941490 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:29.941400    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aacf1f51772e84e5b2b0353a1521470408a11a830a540c00765118de582a1fcc"} err="failed to get container status \"aacf1f51772e84e5b2b0353a1521470408a11a830a540c00765118de582a1fcc\": rpc error: code = NotFound desc = could not find container \"aacf1f51772e84e5b2b0353a1521470408a11a830a540c00765118de582a1fcc\": container with ID starting with aacf1f51772e84e5b2b0353a1521470408a11a830a540c00765118de582a1fcc not found: ID does not exist"
Nov 01 06:46:30.098306 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:30.098274    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"]
Nov 01 06:46:30.104439 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:30.104412    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f73e89-56b9596757-wbl2m"]
Nov 01 06:46:30.334590 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:30.334501    5619 patch_prober.go:29] interesting pod/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 01 06:46:30.334590 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:30.334552    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f" podUID="96534e12-7487-48ec-bfaa-9d7cc14181d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 01 06:46:30.678067 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:30.678024    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:30.970666 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:30.970572    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7e166f7-3bf1-4d53-b9e8-e964a5546a8d" path="/var/lib/kubelet/pods/d7e166f7-3bf1-4d53-b9e8-e964a5546a8d/volumes"
Nov 01 06:46:31.156774 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:31.156729    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:31.664068 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:31.664030    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:31.664705 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:31.664641    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:31.666024 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:31.666004    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:46:31.964612 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:31.964493    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:32.151732 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:32.151693    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:32.643744 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:32.643714    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:33.503537 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:33.503498    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:34.013343 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:34.013286    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:34.035305 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:34.035270    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"]
Nov 01 06:46:34.035556 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:34.035532    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl" podUID="b48e404e-08bf-4a58-84f8-3c095893759f" containerName="istio-proxy" containerID="cri-o://9af6a30638ce8f56d68891aaf143bce408c30af45a6be17ca035bf317abd3320" gracePeriod=30
Nov 01 06:46:36.200848 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:36.200813    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:36.433956 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:36.433918    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:37.056211 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:37.056179    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:39.285647 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.285619    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 06:46:39.300528 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.300503    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b48e404e-08bf-4a58-84f8-3c095893759f-kube-api-access-trmbj\" (UniqueName: \"kubernetes.io/projected/b48e404e-08bf-4a58-84f8-3c095893759f-kube-api-access-trmbj\") pod \"b48e404e-08bf-4a58-84f8-3c095893759f\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") "
Nov 01 06:46:39.300634 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.300542    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b48e404e-08bf-4a58-84f8-3c095893759f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48e404e-08bf-4a58-84f8-3c095893759f-istio-podinfo\") pod \"b48e404e-08bf-4a58-84f8-3c095893759f\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") "
Nov 01 06:46:39.300634 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.300612    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-istio-envoy\") pod \"b48e404e-08bf-4a58-84f8-3c095893759f\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") "
Nov 01 06:46:39.300634 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.300636    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b48e404e-08bf-4a58-84f8-3c095893759f-istio-token\" (UniqueName: \"kubernetes.io/projected/b48e404e-08bf-4a58-84f8-3c095893759f-istio-token\") pod \"b48e404e-08bf-4a58-84f8-3c095893759f\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") "
Nov 01 06:46:39.300821 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.300716    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-workload-socket\") pod \"b48e404e-08bf-4a58-84f8-3c095893759f\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") "
Nov 01 06:46:39.300821 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.300758    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b48e404e-08bf-4a58-84f8-3c095893759f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48e404e-08bf-4a58-84f8-3c095893759f-istiod-ca-cert\") pod \"b48e404e-08bf-4a58-84f8-3c095893759f\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") "
Nov 01 06:46:39.300821 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.300807    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-istio-data\") pod \"b48e404e-08bf-4a58-84f8-3c095893759f\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") "
Nov 01 06:46:39.300968 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.300847    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-credential-socket\") pod \"b48e404e-08bf-4a58-84f8-3c095893759f\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") "
Nov 01 06:46:39.300968 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.300933    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-workload-certs\") pod \"b48e404e-08bf-4a58-84f8-3c095893759f\" (UID: \"b48e404e-08bf-4a58-84f8-3c095893759f\") "
Nov 01 06:46:39.301185 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.301128    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-istio-data" pod "b48e404e-08bf-4a58-84f8-3c095893759f" (UID: "b48e404e-08bf-4a58-84f8-3c095893759f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:39.301299 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.301276    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b48e404e-08bf-4a58-84f8-3c095893759f-istiod-ca-cert" pod "b48e404e-08bf-4a58-84f8-3c095893759f" (UID: "b48e404e-08bf-4a58-84f8-3c095893759f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:46:39.301425 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.301400    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-workload-certs" pod "b48e404e-08bf-4a58-84f8-3c095893759f" (UID: "b48e404e-08bf-4a58-84f8-3c095893759f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:39.301504 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.301444    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-workload-socket" pod "b48e404e-08bf-4a58-84f8-3c095893759f" (UID: "b48e404e-08bf-4a58-84f8-3c095893759f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:39.301504 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.301495    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-credential-socket" pod "b48e404e-08bf-4a58-84f8-3c095893759f" (UID: "b48e404e-08bf-4a58-84f8-3c095893759f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:39.301664 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.301648    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:39.301705 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.301668    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:39.301705 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.301678    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:39.301705 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.301687    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:39.301705 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.301696    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48e404e-08bf-4a58-84f8-3c095893759f-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:39.303230 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.303170    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-istio-envoy" pod "b48e404e-08bf-4a58-84f8-3c095893759f" (UID: "b48e404e-08bf-4a58-84f8-3c095893759f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:46:39.303230 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.303180    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48e404e-08bf-4a58-84f8-3c095893759f-kube-api-access-trmbj" pod "b48e404e-08bf-4a58-84f8-3c095893759f" (UID: "b48e404e-08bf-4a58-84f8-3c095893759f"). InnerVolumeSpecName "kube-api-access-trmbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:46:39.303374 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.303252    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48e404e-08bf-4a58-84f8-3c095893759f-istio-token" pod "b48e404e-08bf-4a58-84f8-3c095893759f" (UID: "b48e404e-08bf-4a58-84f8-3c095893759f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:46:39.303374 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.303318    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b48e404e-08bf-4a58-84f8-3c095893759f-istio-podinfo" pod "b48e404e-08bf-4a58-84f8-3c095893759f" (UID: "b48e404e-08bf-4a58-84f8-3c095893759f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:46:39.402837 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.402793    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trmbj\" (UniqueName: \"kubernetes.io/projected/b48e404e-08bf-4a58-84f8-3c095893759f-kube-api-access-trmbj\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:39.402837 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.402834    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48e404e-08bf-4a58-84f8-3c095893759f-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:39.402837 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.402845    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48e404e-08bf-4a58-84f8-3c095893759f-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:39.403055 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.402854    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48e404e-08bf-4a58-84f8-3c095893759f-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:46:39.995536 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.995497    5619 generic.go:356] "Generic (PLEG): container finished" podID="b48e404e-08bf-4a58-84f8-3c095893759f" containerID="9af6a30638ce8f56d68891aaf143bce408c30af45a6be17ca035bf317abd3320" exitCode=0
Nov 01 06:46:39.995711 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.995547    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"
Nov 01 06:46:39.995711 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.995574    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl" event={"ID":"b48e404e-08bf-4a58-84f8-3c095893759f","Type":"ContainerDied","Data":"9af6a30638ce8f56d68891aaf143bce408c30af45a6be17ca035bf317abd3320"}
Nov 01 06:46:39.995711 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.995662    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl" event={"ID":"b48e404e-08bf-4a58-84f8-3c095893759f","Type":"ContainerDied","Data":"5394445e8166ac89345398e78e10c0c041f56583b3c35b374f67de009eb9d19f"}
Nov 01 06:46:39.995711 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:39.995687    5619 scope.go:122] "RemoveContainer" containerID="9af6a30638ce8f56d68891aaf143bce408c30af45a6be17ca035bf317abd3320"
Nov 01 06:46:40.006034 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:40.005913    5619 scope.go:122] "RemoveContainer" containerID="de2c553c27c69ac68e6669383e58d9ea1eeff584ccf94154de9d756a5c592682"
Nov 01 06:46:40.014845 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:40.014830    5619 scope.go:122] "RemoveContainer" containerID="9af6a30638ce8f56d68891aaf143bce408c30af45a6be17ca035bf317abd3320"
Nov 01 06:46:40.015115 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:46:40.015092    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af6a30638ce8f56d68891aaf143bce408c30af45a6be17ca035bf317abd3320\": container with ID starting with 9af6a30638ce8f56d68891aaf143bce408c30af45a6be17ca035bf317abd3320 not found: ID does not exist" containerID="9af6a30638ce8f56d68891aaf143bce408c30af45a6be17ca035bf317abd3320"
Nov 01 06:46:40.015155 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:40.015124    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af6a30638ce8f56d68891aaf143bce408c30af45a6be17ca035bf317abd3320"} err="failed to get container status \"9af6a30638ce8f56d68891aaf143bce408c30af45a6be17ca035bf317abd3320\": rpc error: code = NotFound desc = could not find container \"9af6a30638ce8f56d68891aaf143bce408c30af45a6be17ca035bf317abd3320\": container with ID starting with 9af6a30638ce8f56d68891aaf143bce408c30af45a6be17ca035bf317abd3320 not found: ID does not exist"
Nov 01 06:46:40.015155 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:40.015143    5619 scope.go:122] "RemoveContainer" containerID="de2c553c27c69ac68e6669383e58d9ea1eeff584ccf94154de9d756a5c592682"
Nov 01 06:46:40.015399 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:46:40.015379    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2c553c27c69ac68e6669383e58d9ea1eeff584ccf94154de9d756a5c592682\": container with ID starting with de2c553c27c69ac68e6669383e58d9ea1eeff584ccf94154de9d756a5c592682 not found: ID does not exist" containerID="de2c553c27c69ac68e6669383e58d9ea1eeff584ccf94154de9d756a5c592682"
Nov 01 06:46:40.015440 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:40.015409    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2c553c27c69ac68e6669383e58d9ea1eeff584ccf94154de9d756a5c592682"} err="failed to get container status \"de2c553c27c69ac68e6669383e58d9ea1eeff584ccf94154de9d756a5c592682\": rpc error: code = NotFound desc = could not find container \"de2c553c27c69ac68e6669383e58d9ea1eeff584ccf94154de9d756a5c592682\": container with ID starting with de2c553c27c69ac68e6669383e58d9ea1eeff584ccf94154de9d756a5c592682 not found: ID does not exist"
Nov 01 06:46:40.214727 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:40.214696    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"]
Nov 01 06:46:40.266929 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:40.266856    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f73e89-openshift-default-fcc7c6dbf-kvpdl"]
Nov 01 06:46:40.963380 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:40.963347    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b48e404e-08bf-4a58-84f8-3c095893759f" path="/var/lib/kubelet/pods/b48e404e-08bf-4a58-84f8-3c095893759f/volumes"
Nov 01 06:46:41.160988 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:41.160956    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:45.954159 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:45.954120    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:47.931086 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:47.931030    5619 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal" podUID="ef0f634a7979920e9df606f3bb474aac" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Nov 01 06:46:50.806341 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:50.806279    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:55.458110 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:46:55.458062    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:46:55.879918 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:46:55.879881    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:47:00.156105 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:00.156063    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:00.709678 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:00.709646    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:00.849473 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:00.849438    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:03.758012 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:03.757965    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:04.947947 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:04.947904    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:09.664637 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:09.664598    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:14.372950 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:14.372909    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:18.812421 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:18.812379    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"]
Nov 01 06:47:18.812983 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:18.812969    5619 cpu_manager.go:414] "containerMap: removing container" podUID="d7e166f7-3bf1-4d53-b9e8-e964a5546a8d" containerName="maas-api"
Nov 01 06:47:18.813033 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:18.812984    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e166f7-3bf1-4d53-b9e8-e964a5546a8d" containerName="maas-api"
Nov 01 06:47:18.813033 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:18.813002    5619 cpu_manager.go:414] "containerMap: removing container" podUID="b48e404e-08bf-4a58-84f8-3c095893759f" containerName="istio-proxy"
Nov 01 06:47:18.813033 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:18.813016    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b48e404e-08bf-4a58-84f8-3c095893759f" containerName="istio-proxy"
Nov 01 06:47:18.813033 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:18.813028    5619 cpu_manager.go:414] "containerMap: removing container" podUID="b48e404e-08bf-4a58-84f8-3c095893759f" containerName="istio-proxy"
Nov 01 06:47:18.813213 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:18.813035    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b48e404e-08bf-4a58-84f8-3c095893759f" containerName="istio-proxy"
Nov 01 06:47:18.813213 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:18.813139    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e166f7-3bf1-4d53-b9e8-e964a5546a8d" containerName="maas-api"
Nov 01 06:47:18.813213 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:18.813151    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="b48e404e-08bf-4a58-84f8-3c095893759f" containerName="istio-proxy"
Nov 01 06:47:18.813213 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:18.813157    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="b48e404e-08bf-4a58-84f8-3c095893759f" containerName="istio-proxy"
Nov 01 06:47:18.816631 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:18.816610    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"
Nov 01 06:47:18.872422 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:18.872378    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-d2260d-serving-cert\""
Nov 01 06:47:18.925693 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:18.925661    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"]
Nov 01 06:47:18.971509 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:18.971477    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f94020-a328-405d-8604-81102600dbb2-metrics-certs\") pod \"maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx\" (UID: \"21f94020-a328-405d-8604-81102600dbb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"
Nov 01 06:47:18.971675 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:18.971552    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxtdg\" (UniqueName: \"kubernetes.io/projected/21f94020-a328-405d-8604-81102600dbb2-kube-api-access-lxtdg\") pod \"maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx\" (UID: \"21f94020-a328-405d-8604-81102600dbb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"
Nov 01 06:47:18.971675 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:18.971630    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f94020-a328-405d-8604-81102600dbb2-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx\" (UID: \"21f94020-a328-405d-8604-81102600dbb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"
Nov 01 06:47:18.971771 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:18.971674    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f94020-a328-405d-8604-81102600dbb2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx\" (UID: \"21f94020-a328-405d-8604-81102600dbb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"
Nov 01 06:47:18.998727 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:18.998694    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:19.073226 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:19.073138    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f94020-a328-405d-8604-81102600dbb2-metrics-certs\") pod \"maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx\" (UID: \"21f94020-a328-405d-8604-81102600dbb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"
Nov 01 06:47:19.073226 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:19.073223    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxtdg\" (UniqueName: \"kubernetes.io/projected/21f94020-a328-405d-8604-81102600dbb2-kube-api-access-lxtdg\") pod \"maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx\" (UID: \"21f94020-a328-405d-8604-81102600dbb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"
Nov 01 06:47:19.073472 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:19.073254    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f94020-a328-405d-8604-81102600dbb2-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx\" (UID: \"21f94020-a328-405d-8604-81102600dbb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"
Nov 01 06:47:19.073472 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:19.073280    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f94020-a328-405d-8604-81102600dbb2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx\" (UID: \"21f94020-a328-405d-8604-81102600dbb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"
Nov 01 06:47:19.073555 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:47:19.073519    5619 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-serving-cert: secret "maas-api-e2e-extmod-b-d2260d-serving-cert" not found
Nov 01 06:47:19.073600 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:47:19.073588    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f94020-a328-405d-8604-81102600dbb2-maas-api-tls podName:21f94020-a328-405d-8604-81102600dbb2 nodeName:}" failed. No retries permitted until 2026-11-01 06:47:19.573571073 +0000 UTC m=+3676.784394565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21f94020-a328-405d-8604-81102600dbb2-maas-api-tls") pod "maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx" (UID: "21f94020-a328-405d-8604-81102600dbb2") : secret "maas-api-e2e-extmod-b-d2260d-serving-cert" not found
Nov 01 06:47:19.074148 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:19.074052    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f94020-a328-405d-8604-81102600dbb2-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx\" (UID: \"21f94020-a328-405d-8604-81102600dbb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"
Nov 01 06:47:19.075945 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:19.075917    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f94020-a328-405d-8604-81102600dbb2-metrics-certs\") pod \"maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx\" (UID: \"21f94020-a328-405d-8604-81102600dbb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"
Nov 01 06:47:19.137032 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:19.137000    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxtdg\" (UniqueName: \"kubernetes.io/projected/21f94020-a328-405d-8604-81102600dbb2-kube-api-access-lxtdg\") pod \"maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx\" (UID: \"21f94020-a328-405d-8604-81102600dbb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"
Nov 01 06:47:19.186785 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:19.186747    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:19.328896 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:19.328798    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:19.578996 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:19.578913    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f94020-a328-405d-8604-81102600dbb2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx\" (UID: \"21f94020-a328-405d-8604-81102600dbb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"
Nov 01 06:47:19.581236 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:19.581208    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f94020-a328-405d-8604-81102600dbb2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx\" (UID: \"21f94020-a328-405d-8604-81102600dbb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"
Nov 01 06:47:19.730565 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:19.730534    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"
Nov 01 06:47:20.107089 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:20.107059    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"]
Nov 01 06:47:20.109403 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:47:20.109374    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f94020_a328_405d_8604_81102600dbb2.slice/crio-fdc0de3d98ab394d49442b6dc6b3b14789e72d73c3b3ccdd19850b350e10f7f6 WatchSource:0}: Error finding container fdc0de3d98ab394d49442b6dc6b3b14789e72d73c3b3ccdd19850b350e10f7f6: Status 404 returned error can't find the container with id fdc0de3d98ab394d49442b6dc6b3b14789e72d73c3b3ccdd19850b350e10f7f6
Nov 01 06:47:20.243730 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:20.243694    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx" event={"ID":"21f94020-a328-405d-8604-81102600dbb2","Type":"ContainerStarted","Data":"fdc0de3d98ab394d49442b6dc6b3b14789e72d73c3b3ccdd19850b350e10f7f6"}
Nov 01 06:47:20.819183 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:20.819141    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:20.958493 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:20.958449    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:21.251803 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:21.251762    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx" event={"ID":"21f94020-a328-405d-8604-81102600dbb2","Type":"ContainerStarted","Data":"4747ac46d16fae685ebacf42c779db5da9a2c5de5aaca71e88805ab106e1cd8a"}
Nov 01 06:47:21.252186 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:21.251841    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"
Nov 01 06:47:21.373075 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:21.373018    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx" podStartSLOduration=2.493584682 podStartE2EDuration="3.373003345s" totalImagesPullingTime="879.418663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:47:21.370247118 +0000 UTC m=+3678.581070626" watchObservedRunningTime="2026-11-01 06:47:21.373003345 +0000 UTC m=+3678.583826853"
Nov 01 06:47:23.951287 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:23.951246    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:25.880170 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:47:25.880135    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:47:27.265936 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:27.265906    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"
Nov 01 06:47:28.712276 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:28.712240    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:32.999911 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:32.999876    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:33.462922 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:33.462885    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:37.370896 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:37.370862    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"]
Nov 01 06:47:37.377619 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:37.377578    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:37.575974 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:37.575925    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:38.189999 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:38.189966    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:42.982562 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:42.982472    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:47.789930 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:47.789882    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:47.800994 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:47.800964    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"]
Nov 01 06:47:48.062735 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:48.062639    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:48.433420 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:48.433376    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:52.391480 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:52.391439    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t" podUID="c2a12b29-2d63-40e9-8704-099d289a05ec" containerName="main" containerID="cri-o://a26d0ed3fe5c4c5dfee1d5cf5fa4c7570054ed7bbeef569c4d65612b54005904" gracePeriod=45
Nov 01 06:47:52.579433 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:52.579339    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:52.673713 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:52.673688    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:47:52.777284 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:52.777254    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-dshm\") pod \"c2a12b29-2d63-40e9-8704-099d289a05ec\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") "
Nov 01 06:47:52.777476 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:52.777385    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-model-cache\") pod \"c2a12b29-2d63-40e9-8704-099d289a05ec\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") "
Nov 01 06:47:52.777476 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:52.777430    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2a12b29-2d63-40e9-8704-099d289a05ec-tls-certs\" (UniqueName: \"kubernetes.io/secret/c2a12b29-2d63-40e9-8704-099d289a05ec-tls-certs\") pod \"c2a12b29-2d63-40e9-8704-099d289a05ec\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") "
Nov 01 06:47:52.777619 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:52.777480    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-home\" (UniqueName: \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-home\") pod \"c2a12b29-2d63-40e9-8704-099d289a05ec\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") "
Nov 01 06:47:52.777619 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:52.777594    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-tmp-dir\") pod \"c2a12b29-2d63-40e9-8704-099d289a05ec\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") "
Nov 01 06:47:52.777733 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:52.777644    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2a12b29-2d63-40e9-8704-099d289a05ec-kube-api-access-wppj9\" (UniqueName: \"kubernetes.io/projected/c2a12b29-2d63-40e9-8704-099d289a05ec-kube-api-access-wppj9\") pod \"c2a12b29-2d63-40e9-8704-099d289a05ec\" (UID: \"c2a12b29-2d63-40e9-8704-099d289a05ec\") "
Nov 01 06:47:52.777733 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:52.777650    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-model-cache" pod "c2a12b29-2d63-40e9-8704-099d289a05ec" (UID: "c2a12b29-2d63-40e9-8704-099d289a05ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:47:52.777852 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:52.777789    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-home" pod "c2a12b29-2d63-40e9-8704-099d289a05ec" (UID: "c2a12b29-2d63-40e9-8704-099d289a05ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:47:52.778025 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:52.778004    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-tmp-dir" pod "c2a12b29-2d63-40e9-8704-099d289a05ec" (UID: "c2a12b29-2d63-40e9-8704-099d289a05ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:47:52.778257 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:52.778236    5619 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-model-cache\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:52.778382 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:52.778262    5619 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-home\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:52.778382 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:52.778277    5619 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-tmp-dir\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:52.779813 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:52.779786    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2a12b29-2d63-40e9-8704-099d289a05ec-tls-certs" pod "c2a12b29-2d63-40e9-8704-099d289a05ec" (UID: "c2a12b29-2d63-40e9-8704-099d289a05ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:47:52.779978 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:52.779960    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-dshm" pod "c2a12b29-2d63-40e9-8704-099d289a05ec" (UID: "c2a12b29-2d63-40e9-8704-099d289a05ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:47:52.780042 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:52.779993    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a12b29-2d63-40e9-8704-099d289a05ec-kube-api-access-wppj9" pod "c2a12b29-2d63-40e9-8704-099d289a05ec" (UID: "c2a12b29-2d63-40e9-8704-099d289a05ec"). InnerVolumeSpecName "kube-api-access-wppj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:47:52.879751 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:52.879714    5619 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2a12b29-2d63-40e9-8704-099d289a05ec-tls-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:52.879751 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:52.879744    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wppj9\" (UniqueName: \"kubernetes.io/projected/c2a12b29-2d63-40e9-8704-099d289a05ec-kube-api-access-wppj9\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:52.879751 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:52.879757    5619 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2a12b29-2d63-40e9-8704-099d289a05ec-dshm\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:47:53.449151 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:53.449113    5619 generic.go:356] "Generic (PLEG): container finished" podID="c2a12b29-2d63-40e9-8704-099d289a05ec" containerID="a26d0ed3fe5c4c5dfee1d5cf5fa4c7570054ed7bbeef569c4d65612b54005904" exitCode=0
Nov 01 06:47:53.449585 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:53.449177    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"
Nov 01 06:47:53.449585 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:53.449193    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t" event={"ID":"c2a12b29-2d63-40e9-8704-099d289a05ec","Type":"ContainerDied","Data":"a26d0ed3fe5c4c5dfee1d5cf5fa4c7570054ed7bbeef569c4d65612b54005904"}
Nov 01 06:47:53.449585 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:53.449247    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t" event={"ID":"c2a12b29-2d63-40e9-8704-099d289a05ec","Type":"ContainerDied","Data":"79ded5acfa06658912ca25db50aa2000d22c08faff566348c68256bafb9fdfea"}
Nov 01 06:47:53.449585 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:53.449279    5619 scope.go:122] "RemoveContainer" containerID="a26d0ed3fe5c4c5dfee1d5cf5fa4c7570054ed7bbeef569c4d65612b54005904"
Nov 01 06:47:53.459122 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:53.459100    5619 scope.go:122] "RemoveContainer" containerID="a26d0ed3fe5c4c5dfee1d5cf5fa4c7570054ed7bbeef569c4d65612b54005904"
Nov 01 06:47:53.459423 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:47:53.459399    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a26d0ed3fe5c4c5dfee1d5cf5fa4c7570054ed7bbeef569c4d65612b54005904\": container with ID starting with a26d0ed3fe5c4c5dfee1d5cf5fa4c7570054ed7bbeef569c4d65612b54005904 not found: ID does not exist" containerID="a26d0ed3fe5c4c5dfee1d5cf5fa4c7570054ed7bbeef569c4d65612b54005904"
Nov 01 06:47:53.459555 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:53.459429    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a26d0ed3fe5c4c5dfee1d5cf5fa4c7570054ed7bbeef569c4d65612b54005904"} err="failed to get container status \"a26d0ed3fe5c4c5dfee1d5cf5fa4c7570054ed7bbeef569c4d65612b54005904\": rpc error: code = NotFound desc = could not find container \"a26d0ed3fe5c4c5dfee1d5cf5fa4c7570054ed7bbeef569c4d65612b54005904\": container with ID starting with a26d0ed3fe5c4c5dfee1d5cf5fa4c7570054ed7bbeef569c4d65612b54005904 not found: ID does not exist"
Nov 01 06:47:53.619131 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:53.619098    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"]
Nov 01 06:47:53.669653 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:53.669619    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-b-be936817-kserve-7c7c586968-mhb5t"]
Nov 01 06:47:54.964733 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:54.964698    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2a12b29-2d63-40e9-8704-099d289a05ec" path="/var/lib/kubelet/pods/c2a12b29-2d63-40e9-8704-099d289a05ec/volumes"
Nov 01 06:47:55.205837 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:55.205802    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:55.885853 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:47:55.885815    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:47:56.122375 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:56.122316    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:56.267850 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:56.267755    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:47:57.509316 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:47:57.509281    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:00.838848 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:00.838813    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:02.399667 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:02.399629    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:02.821130 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:02.821029    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8" podUID="d5532f87-20d8-4fdf-89ce-28ed181176a4" containerName="main" containerID="cri-o://dd69b0951ff227a19c020cd4063fff1ba8110ca27f4e96a63431d6202e829510" gracePeriod=45
Nov 01 06:48:03.108726 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.108698    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:48:03.201648 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.201611    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-tmp-dir\") pod \"d5532f87-20d8-4fdf-89ce-28ed181176a4\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") "
Nov 01 06:48:03.201840 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.201657    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5532f87-20d8-4fdf-89ce-28ed181176a4-tls-certs\" (UniqueName: \"kubernetes.io/secret/d5532f87-20d8-4fdf-89ce-28ed181176a4-tls-certs\") pod \"d5532f87-20d8-4fdf-89ce-28ed181176a4\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") "
Nov 01 06:48:03.201840 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.201700    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-model-cache\") pod \"d5532f87-20d8-4fdf-89ce-28ed181176a4\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") "
Nov 01 06:48:03.201840 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.201747    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-home\" (UniqueName: \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-home\") pod \"d5532f87-20d8-4fdf-89ce-28ed181176a4\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") "
Nov 01 06:48:03.201840 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.201791    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5532f87-20d8-4fdf-89ce-28ed181176a4-kube-api-access-l9vn6\" (UniqueName: \"kubernetes.io/projected/d5532f87-20d8-4fdf-89ce-28ed181176a4-kube-api-access-l9vn6\") pod \"d5532f87-20d8-4fdf-89ce-28ed181176a4\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") "
Nov 01 06:48:03.201840 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.201838    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-dshm\") pod \"d5532f87-20d8-4fdf-89ce-28ed181176a4\" (UID: \"d5532f87-20d8-4fdf-89ce-28ed181176a4\") "
Nov 01 06:48:03.202131 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.201892    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-tmp-dir" pod "d5532f87-20d8-4fdf-89ce-28ed181176a4" (UID: "d5532f87-20d8-4fdf-89ce-28ed181176a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:48:03.202131 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.201977    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-model-cache" pod "d5532f87-20d8-4fdf-89ce-28ed181176a4" (UID: "d5532f87-20d8-4fdf-89ce-28ed181176a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:48:03.202131 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.202012    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-home" pod "d5532f87-20d8-4fdf-89ce-28ed181176a4" (UID: "d5532f87-20d8-4fdf-89ce-28ed181176a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:48:03.202434 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.202414    5619 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-tmp-dir\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:03.202515 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.202442    5619 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-model-cache\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:03.202515 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.202459    5619 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-home\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:03.203824 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.203791    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5532f87-20d8-4fdf-89ce-28ed181176a4-tls-certs" pod "d5532f87-20d8-4fdf-89ce-28ed181176a4" (UID: "d5532f87-20d8-4fdf-89ce-28ed181176a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:48:03.203934 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.203873    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-dshm" pod "d5532f87-20d8-4fdf-89ce-28ed181176a4" (UID: "d5532f87-20d8-4fdf-89ce-28ed181176a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:48:03.203981 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.203937    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5532f87-20d8-4fdf-89ce-28ed181176a4-kube-api-access-l9vn6" pod "d5532f87-20d8-4fdf-89ce-28ed181176a4" (UID: "d5532f87-20d8-4fdf-89ce-28ed181176a4"). InnerVolumeSpecName "kube-api-access-l9vn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:48:03.303863 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.303831    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9vn6\" (UniqueName: \"kubernetes.io/projected/d5532f87-20d8-4fdf-89ce-28ed181176a4-kube-api-access-l9vn6\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:03.303863 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.303859    5619 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5532f87-20d8-4fdf-89ce-28ed181176a4-dshm\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:03.303863 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.303870    5619 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5532f87-20d8-4fdf-89ce-28ed181176a4-tls-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:03.512835 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.512800    5619 generic.go:356] "Generic (PLEG): container finished" podID="d5532f87-20d8-4fdf-89ce-28ed181176a4" containerID="dd69b0951ff227a19c020cd4063fff1ba8110ca27f4e96a63431d6202e829510" exitCode=0
Nov 01 06:48:03.513243 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.512845    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"
Nov 01 06:48:03.513243 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.512888    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8" event={"ID":"d5532f87-20d8-4fdf-89ce-28ed181176a4","Type":"ContainerDied","Data":"dd69b0951ff227a19c020cd4063fff1ba8110ca27f4e96a63431d6202e829510"}
Nov 01 06:48:03.513243 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.512933    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8" event={"ID":"d5532f87-20d8-4fdf-89ce-28ed181176a4","Type":"ContainerDied","Data":"2ceef1fe17e9a33a65f58f5129e62cc5ada7ceea6fdead3c3f873df5005aba04"}
Nov 01 06:48:03.513243 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.512953    5619 scope.go:122] "RemoveContainer" containerID="dd69b0951ff227a19c020cd4063fff1ba8110ca27f4e96a63431d6202e829510"
Nov 01 06:48:03.523598 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.523582    5619 scope.go:122] "RemoveContainer" containerID="dd69b0951ff227a19c020cd4063fff1ba8110ca27f4e96a63431d6202e829510"
Nov 01 06:48:03.523874 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:48:03.523853    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd69b0951ff227a19c020cd4063fff1ba8110ca27f4e96a63431d6202e829510\": container with ID starting with dd69b0951ff227a19c020cd4063fff1ba8110ca27f4e96a63431d6202e829510 not found: ID does not exist" containerID="dd69b0951ff227a19c020cd4063fff1ba8110ca27f4e96a63431d6202e829510"
Nov 01 06:48:03.523944 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.523886    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd69b0951ff227a19c020cd4063fff1ba8110ca27f4e96a63431d6202e829510"} err="failed to get container status \"dd69b0951ff227a19c020cd4063fff1ba8110ca27f4e96a63431d6202e829510\": rpc error: code = NotFound desc = could not find container \"dd69b0951ff227a19c020cd4063fff1ba8110ca27f4e96a63431d6202e829510\": container with ID starting with dd69b0951ff227a19c020cd4063fff1ba8110ca27f4e96a63431d6202e829510 not found: ID does not exist"
Nov 01 06:48:03.679680 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.679646    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"]
Nov 01 06:48:03.730485 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:03.730448    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-14a6e4/e2e-conflict-a-be936817-kserve-5bb465b96b-d4zz8"]
Nov 01 06:48:04.963276 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:04.963239    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5532f87-20d8-4fdf-89ce-28ed181176a4" path="/var/lib/kubelet/pods/d5532f87-20d8-4fdf-89ce-28ed181176a4/volumes"
Nov 01 06:48:07.168544 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:07.168503    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:11.868744 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:11.868698    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:16.666645 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:16.666608    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:21.259131 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:21.259091    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:25.880196 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:48:25.880152    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:48:26.045311 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:26.045274    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:29.935531 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:29.935496    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:30.932808 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:30.932767    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:31.079722 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:31.079686    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:31.383370 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:31.383334    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:31.611300 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:31.611259    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:33.357852 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:33.357816    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:35.107261 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:35.107228    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:35.645391 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:35.645358    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:37.759307 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:37.759265    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:40.379142 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:40.379100    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:42.426961 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:42.426923    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:42.576114 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:42.576077    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:44.390441 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.390406    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"]
Nov 01 06:48:44.390834 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.390610    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c" podUID="5593ec73-2b3f-4d24-b3f9-e142194caff4" containerName="maas-api" containerID="cri-o://1eb4daca213f62618408aae054f84ae3ec3de39213ad17b63d8303843feb176b" gracePeriod=30
Nov 01 06:48:44.492484 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.492448    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:44.656678 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.656621    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"
Nov 01 06:48:44.684703 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.684676    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5593ec73-2b3f-4d24-b3f9-e142194caff4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5593ec73-2b3f-4d24-b3f9-e142194caff4-metrics-certs\") pod \"5593ec73-2b3f-4d24-b3f9-e142194caff4\" (UID: \"5593ec73-2b3f-4d24-b3f9-e142194caff4\") "
Nov 01 06:48:44.684853 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.684723    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5593ec73-2b3f-4d24-b3f9-e142194caff4-kube-api-access-cmvwt\" (UniqueName: \"kubernetes.io/projected/5593ec73-2b3f-4d24-b3f9-e142194caff4-kube-api-access-cmvwt\") pod \"5593ec73-2b3f-4d24-b3f9-e142194caff4\" (UID: \"5593ec73-2b3f-4d24-b3f9-e142194caff4\") "
Nov 01 06:48:44.684853 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.684765    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5593ec73-2b3f-4d24-b3f9-e142194caff4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5593ec73-2b3f-4d24-b3f9-e142194caff4-openshift-service-ca\") pod \"5593ec73-2b3f-4d24-b3f9-e142194caff4\" (UID: \"5593ec73-2b3f-4d24-b3f9-e142194caff4\") "
Nov 01 06:48:44.684853 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.684838    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5593ec73-2b3f-4d24-b3f9-e142194caff4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5593ec73-2b3f-4d24-b3f9-e142194caff4-maas-api-tls\") pod \"5593ec73-2b3f-4d24-b3f9-e142194caff4\" (UID: \"5593ec73-2b3f-4d24-b3f9-e142194caff4\") "
Nov 01 06:48:44.685186 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.685162    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5593ec73-2b3f-4d24-b3f9-e142194caff4-openshift-service-ca" pod "5593ec73-2b3f-4d24-b3f9-e142194caff4" (UID: "5593ec73-2b3f-4d24-b3f9-e142194caff4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:48:44.685279 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.685252    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5593ec73-2b3f-4d24-b3f9-e142194caff4-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:44.686928 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.686899    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5593ec73-2b3f-4d24-b3f9-e142194caff4-metrics-certs" pod "5593ec73-2b3f-4d24-b3f9-e142194caff4" (UID: "5593ec73-2b3f-4d24-b3f9-e142194caff4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:48:44.687042 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.686964    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5593ec73-2b3f-4d24-b3f9-e142194caff4-maas-api-tls" pod "5593ec73-2b3f-4d24-b3f9-e142194caff4" (UID: "5593ec73-2b3f-4d24-b3f9-e142194caff4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:48:44.687042 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.686993    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5593ec73-2b3f-4d24-b3f9-e142194caff4-kube-api-access-cmvwt" pod "5593ec73-2b3f-4d24-b3f9-e142194caff4" (UID: "5593ec73-2b3f-4d24-b3f9-e142194caff4"). InnerVolumeSpecName "kube-api-access-cmvwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:48:44.754095 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.754067    5619 generic.go:356] "Generic (PLEG): container finished" podID="5593ec73-2b3f-4d24-b3f9-e142194caff4" containerID="1eb4daca213f62618408aae054f84ae3ec3de39213ad17b63d8303843feb176b" exitCode=0
Nov 01 06:48:44.754231 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.754135    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"
Nov 01 06:48:44.754231 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.754150    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c" event={"ID":"5593ec73-2b3f-4d24-b3f9-e142194caff4","Type":"ContainerDied","Data":"1eb4daca213f62618408aae054f84ae3ec3de39213ad17b63d8303843feb176b"}
Nov 01 06:48:44.754231 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.754202    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c" event={"ID":"5593ec73-2b3f-4d24-b3f9-e142194caff4","Type":"ContainerDied","Data":"622c43c668ba5081f93a3950d53945af3c9c1fde3e0d4f674f9324eedc62ac3c"}
Nov 01 06:48:44.754231 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.754224    5619 scope.go:122] "RemoveContainer" containerID="1eb4daca213f62618408aae054f84ae3ec3de39213ad17b63d8303843feb176b"
Nov 01 06:48:44.764750 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.764729    5619 scope.go:122] "RemoveContainer" containerID="1eb4daca213f62618408aae054f84ae3ec3de39213ad17b63d8303843feb176b"
Nov 01 06:48:44.765112 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:48:44.765008    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eb4daca213f62618408aae054f84ae3ec3de39213ad17b63d8303843feb176b\": container with ID starting with 1eb4daca213f62618408aae054f84ae3ec3de39213ad17b63d8303843feb176b not found: ID does not exist" containerID="1eb4daca213f62618408aae054f84ae3ec3de39213ad17b63d8303843feb176b"
Nov 01 06:48:44.765112 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.765047    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb4daca213f62618408aae054f84ae3ec3de39213ad17b63d8303843feb176b"} err="failed to get container status \"1eb4daca213f62618408aae054f84ae3ec3de39213ad17b63d8303843feb176b\": rpc error: code = NotFound desc = could not find container \"1eb4daca213f62618408aae054f84ae3ec3de39213ad17b63d8303843feb176b\": container with ID starting with 1eb4daca213f62618408aae054f84ae3ec3de39213ad17b63d8303843feb176b not found: ID does not exist"
Nov 01 06:48:44.809597 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.809563    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5593ec73-2b3f-4d24-b3f9-e142194caff4-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:44.809597 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.809598    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5593ec73-2b3f-4d24-b3f9-e142194caff4-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:44.809852 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.809617    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmvwt\" (UniqueName: \"kubernetes.io/projected/5593ec73-2b3f-4d24-b3f9-e142194caff4-kube-api-access-cmvwt\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:44.896436 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.896295    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"]
Nov 01 06:48:44.901442 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.901402    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8f45a2-6b58cc48c8-mq52c"]
Nov 01 06:48:44.961727 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:44.961655    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5593ec73-2b3f-4d24-b3f9-e142194caff4" path="/var/lib/kubelet/pods/5593ec73-2b3f-4d24-b3f9-e142194caff4/volumes"
Nov 01 06:48:46.282396 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:46.282357    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:46.485376 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:46.485337    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:47.669082 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.669045    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"]
Nov 01 06:48:47.669925 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.669902    5619 cpu_manager.go:414] "containerMap: removing container" podUID="d5532f87-20d8-4fdf-89ce-28ed181176a4" containerName="main"
Nov 01 06:48:47.670037 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.669927    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5532f87-20d8-4fdf-89ce-28ed181176a4" containerName="main"
Nov 01 06:48:47.670037 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.669980    5619 cpu_manager.go:414] "containerMap: removing container" podUID="c2a12b29-2d63-40e9-8704-099d289a05ec" containerName="main"
Nov 01 06:48:47.670037 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.669989    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2a12b29-2d63-40e9-8704-099d289a05ec" containerName="main"
Nov 01 06:48:47.670037 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.670012    5619 cpu_manager.go:414] "containerMap: removing container" podUID="5593ec73-2b3f-4d24-b3f9-e142194caff4" containerName="maas-api"
Nov 01 06:48:47.670037 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.670021    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5593ec73-2b3f-4d24-b3f9-e142194caff4" containerName="maas-api"
Nov 01 06:48:47.670289 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.670141    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5532f87-20d8-4fdf-89ce-28ed181176a4" containerName="main"
Nov 01 06:48:47.670289 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.670162    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2a12b29-2d63-40e9-8704-099d289a05ec" containerName="main"
Nov 01 06:48:47.670289 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.670173    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="5593ec73-2b3f-4d24-b3f9-e142194caff4" containerName="maas-api"
Nov 01 06:48:47.675523 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.675498    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"
Nov 01 06:48:47.730616 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.730582    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"]
Nov 01 06:48:47.755544 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.755512    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-627jz\" (UniqueName: \"kubernetes.io/projected/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-kube-api-access-627jz\") pod \"payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r\" (UID: \"4cc4e41a-b8cb-4d9c-a1a6-07c701659943\") " pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"
Nov 01 06:48:47.755696 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.755554    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r\" (UID: \"4cc4e41a-b8cb-4d9c-a1a6-07c701659943\") " pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"
Nov 01 06:48:47.755696 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.755678    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r\" (UID: \"4cc4e41a-b8cb-4d9c-a1a6-07c701659943\") " pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"
Nov 01 06:48:47.755923 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.755902    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r\" (UID: \"4cc4e41a-b8cb-4d9c-a1a6-07c701659943\") " pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"
Nov 01 06:48:47.783594 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.783562    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5be4fa\"/\"payload-processing-external-model-e2e--63ee1396-dockercfg-cpqmm\""
Nov 01 06:48:47.783594 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.783562    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5be4fa\"/\"routing-overlay\""
Nov 01 06:48:47.783808 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.783593    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5be4fa\"/\"payload-processing-external-model-plugins-e2e-extmod-a-5be4fa\""
Nov 01 06:48:47.783808 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.783633    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5be4fa\"/\"kube-root-ca.crt\""
Nov 01 06:48:47.783808 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.783672    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5be4fa\"/\"openshift-service-ca.crt\""
Nov 01 06:48:47.783948 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.783675    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5be4fa\"/\"agc-cred-893b896c95a8\""
Nov 01 06:48:47.856666 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.856633    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-627jz\" (UniqueName: \"kubernetes.io/projected/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-kube-api-access-627jz\") pod \"payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r\" (UID: \"4cc4e41a-b8cb-4d9c-a1a6-07c701659943\") " pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"
Nov 01 06:48:47.856832 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.856672    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r\" (UID: \"4cc4e41a-b8cb-4d9c-a1a6-07c701659943\") " pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"
Nov 01 06:48:47.856832 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.856717    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r\" (UID: \"4cc4e41a-b8cb-4d9c-a1a6-07c701659943\") " pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"
Nov 01 06:48:47.857006 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.856897    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r\" (UID: \"4cc4e41a-b8cb-4d9c-a1a6-07c701659943\") " pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"
Nov 01 06:48:47.857546 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.857517    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r\" (UID: \"4cc4e41a-b8cb-4d9c-a1a6-07c701659943\") " pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"
Nov 01 06:48:47.857747 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.857721    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r\" (UID: \"4cc4e41a-b8cb-4d9c-a1a6-07c701659943\") " pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"
Nov 01 06:48:47.859212 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.859183    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r\" (UID: \"4cc4e41a-b8cb-4d9c-a1a6-07c701659943\") " pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"
Nov 01 06:48:47.918482 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.918449    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-627jz\" (UniqueName: \"kubernetes.io/projected/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-kube-api-access-627jz\") pod \"payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r\" (UID: \"4cc4e41a-b8cb-4d9c-a1a6-07c701659943\") " pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"
Nov 01 06:48:47.994913 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:47.994835    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"
Nov 01 06:48:48.372181 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:48.372149    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"]
Nov 01 06:48:48.372583 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:48:48.372556    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc4e41a_b8cb_4d9c_a1a6_07c701659943.slice/crio-e94a6ec3a5a690596949047fa6e1875b1c47d943eeb6ccbd029273e6c6552ff0 WatchSource:0}: Error finding container e94a6ec3a5a690596949047fa6e1875b1c47d943eeb6ccbd029273e6c6552ff0: Status 404 returned error can't find the container with id e94a6ec3a5a690596949047fa6e1875b1c47d943eeb6ccbd029273e6c6552ff0
Nov 01 06:48:48.792933 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:48.792839    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r" event={"ID":"4cc4e41a-b8cb-4d9c-a1a6-07c701659943","Type":"ContainerStarted","Data":"aca767221d3954981a53f3b2cfcf09c394b8a7dd9235df9f5bddbb591c63dfb3"}
Nov 01 06:48:48.792933 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:48.792905    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r" event={"ID":"4cc4e41a-b8cb-4d9c-a1a6-07c701659943","Type":"ContainerStarted","Data":"e94a6ec3a5a690596949047fa6e1875b1c47d943eeb6ccbd029273e6c6552ff0"}
Nov 01 06:48:48.793366 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:48.793041    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"
Nov 01 06:48:48.912303 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:48.912254    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r" podStartSLOduration=1.9122401500000001 podStartE2EDuration="1.91224015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:48:48.911022737 +0000 UTC m=+3766.121846261" watchObservedRunningTime="2026-11-01 06:48:48.91224015 +0000 UTC m=+3766.123063669"
Nov 01 06:48:51.173026 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:51.172983    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:52.140297 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:52.140261    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:52.336954 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:52.336893    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:53.743311 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:53.743264    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:54.339620 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:54.339585    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"]
Nov 01 06:48:54.339853 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:54.339795    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl" podUID="4482833c-590d-44a1-a485-651970b9abf6" containerName="istio-proxy" containerID="cri-o://a7b937f3a550f7456b7820648e64c82ff01f10c7cdde356876e658bc9863af06" gracePeriod=30
Nov 01 06:48:54.344218 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:54.344185    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:55.082195 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:55.082153    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:55.878837 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:48:55.878803    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:48:56.243156 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:56.243067    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:48:58.833303 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:58.833268    5619 patch_prober.go:29] interesting pod/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 06:48:58.833784 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:58.833338    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl" podUID="4482833c-590d-44a1-a485-651970b9abf6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 06:48:59.606020 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.605955    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:48:59.694665 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.694632    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4482833c-590d-44a1-a485-651970b9abf6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4482833c-590d-44a1-a485-651970b9abf6-istiod-ca-cert\") pod \"4482833c-590d-44a1-a485-651970b9abf6\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") "
Nov 01 06:48:59.694665 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.694671    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-credential-socket\") pod \"4482833c-590d-44a1-a485-651970b9abf6\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") "
Nov 01 06:48:59.694891 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.694713    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-istio-data\") pod \"4482833c-590d-44a1-a485-651970b9abf6\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") "
Nov 01 06:48:59.694891 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.694757    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-workload-socket\") pod \"4482833c-590d-44a1-a485-651970b9abf6\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") "
Nov 01 06:48:59.694891 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.694808    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4482833c-590d-44a1-a485-651970b9abf6-istio-token\" (UniqueName: \"kubernetes.io/projected/4482833c-590d-44a1-a485-651970b9abf6-istio-token\") pod \"4482833c-590d-44a1-a485-651970b9abf6\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") "
Nov 01 06:48:59.694891 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.694860    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-workload-certs\") pod \"4482833c-590d-44a1-a485-651970b9abf6\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") "
Nov 01 06:48:59.695111 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.694902    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4482833c-590d-44a1-a485-651970b9abf6-kube-api-access-vzxxd\" (UniqueName: \"kubernetes.io/projected/4482833c-590d-44a1-a485-651970b9abf6-kube-api-access-vzxxd\") pod \"4482833c-590d-44a1-a485-651970b9abf6\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") "
Nov 01 06:48:59.695111 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.694925    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4482833c-590d-44a1-a485-651970b9abf6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4482833c-590d-44a1-a485-651970b9abf6-istio-podinfo\") pod \"4482833c-590d-44a1-a485-651970b9abf6\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") "
Nov 01 06:48:59.695111 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.695001    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-credential-socket" pod "4482833c-590d-44a1-a485-651970b9abf6" (UID: "4482833c-590d-44a1-a485-651970b9abf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:48:59.695111 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.695028    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4482833c-590d-44a1-a485-651970b9abf6-istiod-ca-cert" pod "4482833c-590d-44a1-a485-651970b9abf6" (UID: "4482833c-590d-44a1-a485-651970b9abf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:48:59.695111 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.695036    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-istio-envoy\") pod \"4482833c-590d-44a1-a485-651970b9abf6\" (UID: \"4482833c-590d-44a1-a485-651970b9abf6\") "
Nov 01 06:48:59.695415 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.695158    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-workload-certs" pod "4482833c-590d-44a1-a485-651970b9abf6" (UID: "4482833c-590d-44a1-a485-651970b9abf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:48:59.695530 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.695506    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-istio-data" pod "4482833c-590d-44a1-a485-651970b9abf6" (UID: "4482833c-590d-44a1-a485-651970b9abf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:48:59.695600 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.695542    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-workload-socket" pod "4482833c-590d-44a1-a485-651970b9abf6" (UID: "4482833c-590d-44a1-a485-651970b9abf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:48:59.695720 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.695694    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:59.695780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.695720    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4482833c-590d-44a1-a485-651970b9abf6-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:59.695780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.695736    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:59.695780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.695751    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:59.695780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.695765    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:59.697299 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.697276    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4482833c-590d-44a1-a485-651970b9abf6-istio-token" pod "4482833c-590d-44a1-a485-651970b9abf6" (UID: "4482833c-590d-44a1-a485-651970b9abf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:48:59.697464 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.697315    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4482833c-590d-44a1-a485-651970b9abf6-istio-podinfo" pod "4482833c-590d-44a1-a485-651970b9abf6" (UID: "4482833c-590d-44a1-a485-651970b9abf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:48:59.697533 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.697492    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-istio-envoy" pod "4482833c-590d-44a1-a485-651970b9abf6" (UID: "4482833c-590d-44a1-a485-651970b9abf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:48:59.697588 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.697555    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4482833c-590d-44a1-a485-651970b9abf6-kube-api-access-vzxxd" pod "4482833c-590d-44a1-a485-651970b9abf6" (UID: "4482833c-590d-44a1-a485-651970b9abf6"). InnerVolumeSpecName "kube-api-access-vzxxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:48:59.797157 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.797121    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4482833c-590d-44a1-a485-651970b9abf6-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:59.797157 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.797160    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzxxd\" (UniqueName: \"kubernetes.io/projected/4482833c-590d-44a1-a485-651970b9abf6-kube-api-access-vzxxd\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:59.797436 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.797177    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4482833c-590d-44a1-a485-651970b9abf6-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:59.797436 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.797192    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4482833c-590d-44a1-a485-651970b9abf6-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:48:59.802574 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.802540    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"
Nov 01 06:48:59.858804 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.858721    5619 generic.go:356] "Generic (PLEG): container finished" podID="4482833c-590d-44a1-a485-651970b9abf6" containerID="a7b937f3a550f7456b7820648e64c82ff01f10c7cdde356876e658bc9863af06" exitCode=0
Nov 01 06:48:59.858804 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.858784    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"
Nov 01 06:48:59.859288 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.858816    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl" event={"ID":"4482833c-590d-44a1-a485-651970b9abf6","Type":"ContainerDied","Data":"a7b937f3a550f7456b7820648e64c82ff01f10c7cdde356876e658bc9863af06"}
Nov 01 06:48:59.859288 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.858881    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl" event={"ID":"4482833c-590d-44a1-a485-651970b9abf6","Type":"ContainerDied","Data":"46923d1a70ce5980e89b633d70d0ded87d73c9382d11b1970a7767fd9f885c20"}
Nov 01 06:48:59.859288 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.858914    5619 scope.go:122] "RemoveContainer" containerID="a7b937f3a550f7456b7820648e64c82ff01f10c7cdde356876e658bc9863af06"
Nov 01 06:48:59.868771 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.868755    5619 scope.go:122] "RemoveContainer" containerID="4c6426a84a6077a5ae7e36ce85a8f77889a878a31b88eb5c96419fa2a96db924"
Nov 01 06:48:59.877829 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.877811    5619 scope.go:122] "RemoveContainer" containerID="a7b937f3a550f7456b7820648e64c82ff01f10c7cdde356876e658bc9863af06"
Nov 01 06:48:59.878092 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:48:59.878073    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b937f3a550f7456b7820648e64c82ff01f10c7cdde356876e658bc9863af06\": container with ID starting with a7b937f3a550f7456b7820648e64c82ff01f10c7cdde356876e658bc9863af06 not found: ID does not exist" containerID="a7b937f3a550f7456b7820648e64c82ff01f10c7cdde356876e658bc9863af06"
Nov 01 06:48:59.878154 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.878100    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b937f3a550f7456b7820648e64c82ff01f10c7cdde356876e658bc9863af06"} err="failed to get container status \"a7b937f3a550f7456b7820648e64c82ff01f10c7cdde356876e658bc9863af06\": rpc error: code = NotFound desc = could not find container \"a7b937f3a550f7456b7820648e64c82ff01f10c7cdde356876e658bc9863af06\": container with ID starting with a7b937f3a550f7456b7820648e64c82ff01f10c7cdde356876e658bc9863af06 not found: ID does not exist"
Nov 01 06:48:59.878154 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.878126    5619 scope.go:122] "RemoveContainer" containerID="4c6426a84a6077a5ae7e36ce85a8f77889a878a31b88eb5c96419fa2a96db924"
Nov 01 06:48:59.878365 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:48:59.878347    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c6426a84a6077a5ae7e36ce85a8f77889a878a31b88eb5c96419fa2a96db924\": container with ID starting with 4c6426a84a6077a5ae7e36ce85a8f77889a878a31b88eb5c96419fa2a96db924 not found: ID does not exist" containerID="4c6426a84a6077a5ae7e36ce85a8f77889a878a31b88eb5c96419fa2a96db924"
Nov 01 06:48:59.878420 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:48:59.878368    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c6426a84a6077a5ae7e36ce85a8f77889a878a31b88eb5c96419fa2a96db924"} err="failed to get container status \"4c6426a84a6077a5ae7e36ce85a8f77889a878a31b88eb5c96419fa2a96db924\": rpc error: code = NotFound desc = could not find container \"4c6426a84a6077a5ae7e36ce85a8f77889a878a31b88eb5c96419fa2a96db924\": container with ID starting with 4c6426a84a6077a5ae7e36ce85a8f77889a878a31b88eb5c96419fa2a96db924 not found: ID does not exist"
Nov 01 06:49:00.035662 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:00.035631    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"]
Nov 01 06:49:00.039883 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:00.039856    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-8f45a2-openshift-default-5769c96f97-225nl"]
Nov 01 06:49:00.961780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:00.961741    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4482833c-590d-44a1-a485-651970b9abf6" path="/var/lib/kubelet/pods/4482833c-590d-44a1-a485-651970b9abf6/volumes"
Nov 01 06:49:00.980987 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:00.980951    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:01.181134 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:01.181103    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:01.758894 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:01.758854    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c"]
Nov 01 06:49:02.011123 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:02.011031    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:02.205844 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:02.205797    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:06.167118 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:06.167072    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:06.622613 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:06.622571    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:06.759455 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:06.759408    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c" podUID="e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6" containerName="payload-pre-processing" containerID="cri-o://7086e5794e23a5387412c1ab877417d0aaad7b97eafbcbd00039d4af9ff51eb2" gracePeriod=25
Nov 01 06:49:10.284890 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:10.284861    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:12.068425 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:12.068305    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:12.464810 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:12.464758    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:12.616091 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:12.616060    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:13.838723 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:13.838673    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c" podUID="e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:49:14.023947 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:14.023912    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:14.478674 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:14.478637    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:14.930840 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:14.930804    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:17.092070 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:17.092033    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:19.983610 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:19.983570    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:21.675188 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:21.675150    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:21.980034 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:21.979931    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:23.839090 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:23.839050    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c" podUID="e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:49:25.883179 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:49:25.883138    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:49:26.728946 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:26.728891    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:27.032572 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:27.032546    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c"
Nov 01 06:49:27.034565 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:27.034540    5619 generic.go:356] "Generic (PLEG): container finished" podID="e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6" containerID="7086e5794e23a5387412c1ab877417d0aaad7b97eafbcbd00039d4af9ff51eb2" exitCode=0
Nov 01 06:49:27.034693 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:27.034580    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c"
Nov 01 06:49:27.034693 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:27.034625    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c" event={"ID":"e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6","Type":"ContainerDied","Data":"7086e5794e23a5387412c1ab877417d0aaad7b97eafbcbd00039d4af9ff51eb2"}
Nov 01 06:49:27.034693 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:27.034684    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c" event={"ID":"e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6","Type":"ContainerDied","Data":"c9a0b6e8c7114f8978da2e6ad022a96f056634204fd1eb56d14154fb120ea266"}
Nov 01 06:49:27.034801 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:27.034709    5619 scope.go:122] "RemoveContainer" containerID="7086e5794e23a5387412c1ab877417d0aaad7b97eafbcbd00039d4af9ff51eb2"
Nov 01 06:49:27.043529 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:27.043515    5619 scope.go:122] "RemoveContainer" containerID="7086e5794e23a5387412c1ab877417d0aaad7b97eafbcbd00039d4af9ff51eb2"
Nov 01 06:49:27.043793 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:49:27.043774    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7086e5794e23a5387412c1ab877417d0aaad7b97eafbcbd00039d4af9ff51eb2\": container with ID starting with 7086e5794e23a5387412c1ab877417d0aaad7b97eafbcbd00039d4af9ff51eb2 not found: ID does not exist" containerID="7086e5794e23a5387412c1ab877417d0aaad7b97eafbcbd00039d4af9ff51eb2"
Nov 01 06:49:27.043860 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:27.043800    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7086e5794e23a5387412c1ab877417d0aaad7b97eafbcbd00039d4af9ff51eb2"} err="failed to get container status \"7086e5794e23a5387412c1ab877417d0aaad7b97eafbcbd00039d4af9ff51eb2\": rpc error: code = NotFound desc = could not find container \"7086e5794e23a5387412c1ab877417d0aaad7b97eafbcbd00039d4af9ff51eb2\": container with ID starting with 7086e5794e23a5387412c1ab877417d0aaad7b97eafbcbd00039d4af9ff51eb2 not found: ID does not exist"
Nov 01 06:49:27.114997 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:27.114966    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6-kube-api-access-nk48d\" (UniqueName: \"kubernetes.io/projected/e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6-kube-api-access-nk48d\") pod \"e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6\" (UID: \"e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6\") "
Nov 01 06:49:27.115210 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:27.115190    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6-plugins-config-volume\") pod \"e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6\" (UID: \"e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6\") "
Nov 01 06:49:27.115569 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:27.115545    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6-plugins-config-volume" pod "e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6" (UID: "e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:49:27.115709 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:27.115670    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:49:27.117103 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:27.117076    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6-kube-api-access-nk48d" pod "e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6" (UID: "e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6"). InnerVolumeSpecName "kube-api-access-nk48d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:49:27.198797 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:27.198761    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:27.217208 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:27.217183    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk48d\" (UniqueName: \"kubernetes.io/projected/e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6-kube-api-access-nk48d\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:49:27.349581 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:27.349544    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:27.470598 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:27.470561    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c"]
Nov 01 06:49:27.470805 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:27.470626    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7d25db-547bbbc8ff-lbv7c"]
Nov 01 06:49:28.622338 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:28.622286    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:28.965540 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:28.963639    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6" path="/var/lib/kubelet/pods/e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6/volumes"
Nov 01 06:49:29.648756 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:29.648726    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:29.796941 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:29.796901    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:31.666383 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:31.666347    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:32.954952 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:32.954913    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:33.704079 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:33.704040    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:34.888622 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:34.888589    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:36.468930 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:36.468899    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:37.431307 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:37.431269    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:37.625976 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:37.625939    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:38.108020 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:38.107981    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:40.679314 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:40.679280    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:41.264446 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:41.264413    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:46.099934 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:46.099895    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:50.755780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:50.755735    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:55.476292 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:55.476257    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:55.879868 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:49:55.879826    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:49:57.875683 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:57.875641    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:58.072310 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:58.072275    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:58.212163 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:58.212083    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:59.584680 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:59.584650    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"]
Nov 01 06:49:59.585084 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:59.584854    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5" podUID="5fa0499e-e650-4321-9162-ca5b191f1ec5" containerName="maas-api" containerID="cri-o://2522a50ca5f1d2d8561dbd29b50c2f31a98d6defd105298f5e8fcd7eea4a0931" gracePeriod=30
Nov 01 06:49:59.707768 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:59.707732    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:49:59.845513 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:59.845486    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"
Nov 01 06:49:59.983459 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:59.983411    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fa0499e-e650-4321-9162-ca5b191f1ec5-kube-api-access-x59d2\" (UniqueName: \"kubernetes.io/projected/5fa0499e-e650-4321-9162-ca5b191f1ec5-kube-api-access-x59d2\") pod \"5fa0499e-e650-4321-9162-ca5b191f1ec5\" (UID: \"5fa0499e-e650-4321-9162-ca5b191f1ec5\") "
Nov 01 06:49:59.983672 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:59.983486    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fa0499e-e650-4321-9162-ca5b191f1ec5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fa0499e-e650-4321-9162-ca5b191f1ec5-metrics-certs\") pod \"5fa0499e-e650-4321-9162-ca5b191f1ec5\" (UID: \"5fa0499e-e650-4321-9162-ca5b191f1ec5\") "
Nov 01 06:49:59.983810 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:59.983784    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fa0499e-e650-4321-9162-ca5b191f1ec5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fa0499e-e650-4321-9162-ca5b191f1ec5-maas-api-tls\") pod \"5fa0499e-e650-4321-9162-ca5b191f1ec5\" (UID: \"5fa0499e-e650-4321-9162-ca5b191f1ec5\") "
Nov 01 06:49:59.983894 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:59.983840    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fa0499e-e650-4321-9162-ca5b191f1ec5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa0499e-e650-4321-9162-ca5b191f1ec5-openshift-service-ca\") pod \"5fa0499e-e650-4321-9162-ca5b191f1ec5\" (UID: \"5fa0499e-e650-4321-9162-ca5b191f1ec5\") "
Nov 01 06:49:59.986211 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:59.986150    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa0499e-e650-4321-9162-ca5b191f1ec5-openshift-service-ca" pod "5fa0499e-e650-4321-9162-ca5b191f1ec5" (UID: "5fa0499e-e650-4321-9162-ca5b191f1ec5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:49:59.988141 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:59.988109    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa0499e-e650-4321-9162-ca5b191f1ec5-kube-api-access-x59d2" pod "5fa0499e-e650-4321-9162-ca5b191f1ec5" (UID: "5fa0499e-e650-4321-9162-ca5b191f1ec5"). InnerVolumeSpecName "kube-api-access-x59d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:49:59.988275 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:59.988173    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa0499e-e650-4321-9162-ca5b191f1ec5-maas-api-tls" pod "5fa0499e-e650-4321-9162-ca5b191f1ec5" (UID: "5fa0499e-e650-4321-9162-ca5b191f1ec5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:49:59.995655 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:49:59.995627    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa0499e-e650-4321-9162-ca5b191f1ec5-metrics-certs" pod "5fa0499e-e650-4321-9162-ca5b191f1ec5" (UID: "5fa0499e-e650-4321-9162-ca5b191f1ec5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:50:00.086005 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:00.085967    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x59d2\" (UniqueName: \"kubernetes.io/projected/5fa0499e-e650-4321-9162-ca5b191f1ec5-kube-api-access-x59d2\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:00.086005 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:00.086008    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fa0499e-e650-4321-9162-ca5b191f1ec5-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:00.086386 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:00.086025    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fa0499e-e650-4321-9162-ca5b191f1ec5-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:00.086386 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:00.086038    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa0499e-e650-4321-9162-ca5b191f1ec5-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:00.229832 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:00.229754    5619 generic.go:356] "Generic (PLEG): container finished" podID="5fa0499e-e650-4321-9162-ca5b191f1ec5" containerID="2522a50ca5f1d2d8561dbd29b50c2f31a98d6defd105298f5e8fcd7eea4a0931" exitCode=0
Nov 01 06:50:00.229832 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:00.229807    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"
Nov 01 06:50:00.230017 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:00.229842    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5" event={"ID":"5fa0499e-e650-4321-9162-ca5b191f1ec5","Type":"ContainerDied","Data":"2522a50ca5f1d2d8561dbd29b50c2f31a98d6defd105298f5e8fcd7eea4a0931"}
Nov 01 06:50:00.230017 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:00.229897    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5" event={"ID":"5fa0499e-e650-4321-9162-ca5b191f1ec5","Type":"ContainerDied","Data":"a9a82d003ef98a540b3fb4740ed9565351696301fe8313b78359619221829007"}
Nov 01 06:50:00.230017 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:00.229926    5619 scope.go:122] "RemoveContainer" containerID="2522a50ca5f1d2d8561dbd29b50c2f31a98d6defd105298f5e8fcd7eea4a0931"
Nov 01 06:50:00.240283 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:00.240266    5619 scope.go:122] "RemoveContainer" containerID="2522a50ca5f1d2d8561dbd29b50c2f31a98d6defd105298f5e8fcd7eea4a0931"
Nov 01 06:50:00.240559 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:50:00.240539    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2522a50ca5f1d2d8561dbd29b50c2f31a98d6defd105298f5e8fcd7eea4a0931\": container with ID starting with 2522a50ca5f1d2d8561dbd29b50c2f31a98d6defd105298f5e8fcd7eea4a0931 not found: ID does not exist" containerID="2522a50ca5f1d2d8561dbd29b50c2f31a98d6defd105298f5e8fcd7eea4a0931"
Nov 01 06:50:00.240633 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:00.240565    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2522a50ca5f1d2d8561dbd29b50c2f31a98d6defd105298f5e8fcd7eea4a0931"} err="failed to get container status \"2522a50ca5f1d2d8561dbd29b50c2f31a98d6defd105298f5e8fcd7eea4a0931\": rpc error: code = NotFound desc = could not find container \"2522a50ca5f1d2d8561dbd29b50c2f31a98d6defd105298f5e8fcd7eea4a0931\": container with ID starting with 2522a50ca5f1d2d8561dbd29b50c2f31a98d6defd105298f5e8fcd7eea4a0931 not found: ID does not exist"
Nov 01 06:50:00.397128 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:00.397093    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"]
Nov 01 06:50:00.447853 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:00.447821    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-14a6e4-794f49cd5b-255s5"]
Nov 01 06:50:00.962143 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:00.962106    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fa0499e-e650-4321-9162-ca5b191f1ec5" path="/var/lib/kubelet/pods/5fa0499e-e650-4321-9162-ca5b191f1ec5/volumes"
Nov 01 06:50:01.394464 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:01.394430    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:03.938234 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:03.938189    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:04.080546 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:04.080504    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:04.305405 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:04.305300    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:04.449462 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:04.449422    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:05.977967 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:05.977928    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:06.402255 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:06.402212    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:06.509971 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:06.509935    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"]
Nov 01 06:50:06.510203 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:06.510166    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f" podUID="96534e12-7487-48ec-bfaa-9d7cc14181d2" containerName="istio-proxy" containerID="cri-o://7ce9adb6453fa7caa91f215697ef08e0206fb6bf445e170e723b09fddc07b4f9" gracePeriod=30
Nov 01 06:50:06.588148 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:06.588113    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:09.141071 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:09.141034    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"]
Nov 01 06:50:09.141565 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:09.141289    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k" podUID="69cd086b-554c-49c9-a471-831d40610ceb" containerName="istio-proxy" containerID="cri-o://a135903a60c7cfdd5993c3726afbf831edd114bcd689f1f19b7971081968b924" gracePeriod=30
Nov 01 06:50:11.487585 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.487548    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:11.767826 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.767797    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:50:11.918670 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.918633    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-podinfo\") pod \"96534e12-7487-48ec-bfaa-9d7cc14181d2\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") "
Nov 01 06:50:11.918886 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.918685    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-token\" (UniqueName: \"kubernetes.io/projected/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-token\") pod \"96534e12-7487-48ec-bfaa-9d7cc14181d2\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") "
Nov 01 06:50:11.918886 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.918765    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-credential-socket\") pod \"96534e12-7487-48ec-bfaa-9d7cc14181d2\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") "
Nov 01 06:50:11.918886 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.918796    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96534e12-7487-48ec-bfaa-9d7cc14181d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96534e12-7487-48ec-bfaa-9d7cc14181d2-istiod-ca-cert\") pod \"96534e12-7487-48ec-bfaa-9d7cc14181d2\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") "
Nov 01 06:50:11.918886 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.918813    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-envoy\") pod \"96534e12-7487-48ec-bfaa-9d7cc14181d2\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") "
Nov 01 06:50:11.918886 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.918847    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-workload-certs\") pod \"96534e12-7487-48ec-bfaa-9d7cc14181d2\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") "
Nov 01 06:50:11.918886 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.918877    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96534e12-7487-48ec-bfaa-9d7cc14181d2-kube-api-access-5zlqz\" (UniqueName: \"kubernetes.io/projected/96534e12-7487-48ec-bfaa-9d7cc14181d2-kube-api-access-5zlqz\") pod \"96534e12-7487-48ec-bfaa-9d7cc14181d2\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") "
Nov 01 06:50:11.919229 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.918933    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-workload-socket\") pod \"96534e12-7487-48ec-bfaa-9d7cc14181d2\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") "
Nov 01 06:50:11.919229 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.918969    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-data\") pod \"96534e12-7487-48ec-bfaa-9d7cc14181d2\" (UID: \"96534e12-7487-48ec-bfaa-9d7cc14181d2\") "
Nov 01 06:50:11.919229 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.919103    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-credential-socket" pod "96534e12-7487-48ec-bfaa-9d7cc14181d2" (UID: "96534e12-7487-48ec-bfaa-9d7cc14181d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:11.919229 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.919124    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-workload-certs" pod "96534e12-7487-48ec-bfaa-9d7cc14181d2" (UID: "96534e12-7487-48ec-bfaa-9d7cc14181d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:11.919470 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.919275    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-workload-socket" pod "96534e12-7487-48ec-bfaa-9d7cc14181d2" (UID: "96534e12-7487-48ec-bfaa-9d7cc14181d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:11.919470 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.919287    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-data" pod "96534e12-7487-48ec-bfaa-9d7cc14181d2" (UID: "96534e12-7487-48ec-bfaa-9d7cc14181d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:11.919470 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.919356    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96534e12-7487-48ec-bfaa-9d7cc14181d2-istiod-ca-cert" pod "96534e12-7487-48ec-bfaa-9d7cc14181d2" (UID: "96534e12-7487-48ec-bfaa-9d7cc14181d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:50:11.919633 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.919501    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:11.919633 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.919523    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:11.919633 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.919541    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:11.919633 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.919562    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96534e12-7487-48ec-bfaa-9d7cc14181d2-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:11.919633 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.919577    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:11.921042 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.921023    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-token" pod "96534e12-7487-48ec-bfaa-9d7cc14181d2" (UID: "96534e12-7487-48ec-bfaa-9d7cc14181d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:50:11.921203 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.921187    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-envoy" pod "96534e12-7487-48ec-bfaa-9d7cc14181d2" (UID: "96534e12-7487-48ec-bfaa-9d7cc14181d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:11.921316 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.921299    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96534e12-7487-48ec-bfaa-9d7cc14181d2-kube-api-access-5zlqz" pod "96534e12-7487-48ec-bfaa-9d7cc14181d2" (UID: "96534e12-7487-48ec-bfaa-9d7cc14181d2"). InnerVolumeSpecName "kube-api-access-5zlqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:50:11.921393 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:11.921313    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-podinfo" pod "96534e12-7487-48ec-bfaa-9d7cc14181d2" (UID: "96534e12-7487-48ec-bfaa-9d7cc14181d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:50:12.020510 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:12.020434    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:12.020510 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:12.020463    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:12.020510 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:12.020473    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96534e12-7487-48ec-bfaa-9d7cc14181d2-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:12.020510 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:12.020482    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zlqz\" (UniqueName: \"kubernetes.io/projected/96534e12-7487-48ec-bfaa-9d7cc14181d2-kube-api-access-5zlqz\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:12.308308 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:12.308216    5619 generic.go:356] "Generic (PLEG): container finished" podID="96534e12-7487-48ec-bfaa-9d7cc14181d2" containerID="7ce9adb6453fa7caa91f215697ef08e0206fb6bf445e170e723b09fddc07b4f9" exitCode=0
Nov 01 06:50:12.308308 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:12.308272    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"
Nov 01 06:50:12.308562 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:12.308295    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f" event={"ID":"96534e12-7487-48ec-bfaa-9d7cc14181d2","Type":"ContainerDied","Data":"7ce9adb6453fa7caa91f215697ef08e0206fb6bf445e170e723b09fddc07b4f9"}
Nov 01 06:50:12.308562 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:12.308381    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f" event={"ID":"96534e12-7487-48ec-bfaa-9d7cc14181d2","Type":"ContainerDied","Data":"0d821f9a54dd0eb9ff9ddac5f39015697423efa46abe255ac77f08053ffe647d"}
Nov 01 06:50:12.308562 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:12.308404    5619 scope.go:122] "RemoveContainer" containerID="7ce9adb6453fa7caa91f215697ef08e0206fb6bf445e170e723b09fddc07b4f9"
Nov 01 06:50:12.321907 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:12.321886    5619 scope.go:122] "RemoveContainer" containerID="7ce9adb6453fa7caa91f215697ef08e0206fb6bf445e170e723b09fddc07b4f9"
Nov 01 06:50:12.322166 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:50:12.322145    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ce9adb6453fa7caa91f215697ef08e0206fb6bf445e170e723b09fddc07b4f9\": container with ID starting with 7ce9adb6453fa7caa91f215697ef08e0206fb6bf445e170e723b09fddc07b4f9 not found: ID does not exist" containerID="7ce9adb6453fa7caa91f215697ef08e0206fb6bf445e170e723b09fddc07b4f9"
Nov 01 06:50:12.322216 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:12.322178    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ce9adb6453fa7caa91f215697ef08e0206fb6bf445e170e723b09fddc07b4f9"} err="failed to get container status \"7ce9adb6453fa7caa91f215697ef08e0206fb6bf445e170e723b09fddc07b4f9\": rpc error: code = NotFound desc = could not find container \"7ce9adb6453fa7caa91f215697ef08e0206fb6bf445e170e723b09fddc07b4f9\": container with ID starting with 7ce9adb6453fa7caa91f215697ef08e0206fb6bf445e170e723b09fddc07b4f9 not found: ID does not exist"
Nov 01 06:50:12.477500 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:12.477460    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"]
Nov 01 06:50:12.481768 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:12.481730    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-14a6e4-openshift-default-7fccb55f75-jx45f"]
Nov 01 06:50:12.963015 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:12.962981    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96534e12-7487-48ec-bfaa-9d7cc14181d2" path="/var/lib/kubelet/pods/96534e12-7487-48ec-bfaa-9d7cc14181d2/volumes"
Nov 01 06:50:14.324550 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.324510    5619 generic.go:356] "Generic (PLEG): container finished" podID="69cd086b-554c-49c9-a471-831d40610ceb" containerID="a135903a60c7cfdd5993c3726afbf831edd114bcd689f1f19b7971081968b924" exitCode=0
Nov 01 06:50:14.324964 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.324574    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k" event={"ID":"69cd086b-554c-49c9-a471-831d40610ceb","Type":"ContainerDied","Data":"a135903a60c7cfdd5993c3726afbf831edd114bcd689f1f19b7971081968b924"}
Nov 01 06:50:14.404995 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.404967    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:50:14.545645 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.545574    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-istio-data\") pod \"69cd086b-554c-49c9-a471-831d40610ceb\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") "
Nov 01 06:50:14.545645 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.545621    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69cd086b-554c-49c9-a471-831d40610ceb-kube-api-access-g5tng\" (UniqueName: \"kubernetes.io/projected/69cd086b-554c-49c9-a471-831d40610ceb-kube-api-access-g5tng\") pod \"69cd086b-554c-49c9-a471-831d40610ceb\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") "
Nov 01 06:50:14.545832 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.545677    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-istio-envoy\") pod \"69cd086b-554c-49c9-a471-831d40610ceb\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") "
Nov 01 06:50:14.545832 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.545696    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69cd086b-554c-49c9-a471-831d40610ceb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69cd086b-554c-49c9-a471-831d40610ceb-istiod-ca-cert\") pod \"69cd086b-554c-49c9-a471-831d40610ceb\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") "
Nov 01 06:50:14.545832 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.545714    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-credential-socket\") pod \"69cd086b-554c-49c9-a471-831d40610ceb\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") "
Nov 01 06:50:14.545832 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.545752    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69cd086b-554c-49c9-a471-831d40610ceb-istio-token\" (UniqueName: \"kubernetes.io/projected/69cd086b-554c-49c9-a471-831d40610ceb-istio-token\") pod \"69cd086b-554c-49c9-a471-831d40610ceb\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") "
Nov 01 06:50:14.546048 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.545862    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-istio-data" pod "69cd086b-554c-49c9-a471-831d40610ceb" (UID: "69cd086b-554c-49c9-a471-831d40610ceb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:14.546048 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.545919    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-workload-socket\") pod \"69cd086b-554c-49c9-a471-831d40610ceb\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") "
Nov 01 06:50:14.546048 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.545961    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-workload-certs\") pod \"69cd086b-554c-49c9-a471-831d40610ceb\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") "
Nov 01 06:50:14.546048 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.546006    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69cd086b-554c-49c9-a471-831d40610ceb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69cd086b-554c-49c9-a471-831d40610ceb-istio-podinfo\") pod \"69cd086b-554c-49c9-a471-831d40610ceb\" (UID: \"69cd086b-554c-49c9-a471-831d40610ceb\") "
Nov 01 06:50:14.546281 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.546069    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-credential-socket" pod "69cd086b-554c-49c9-a471-831d40610ceb" (UID: "69cd086b-554c-49c9-a471-831d40610ceb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:14.546281 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.546100    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69cd086b-554c-49c9-a471-831d40610ceb-istiod-ca-cert" pod "69cd086b-554c-49c9-a471-831d40610ceb" (UID: "69cd086b-554c-49c9-a471-831d40610ceb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:50:14.546281 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.546192    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-workload-socket" pod "69cd086b-554c-49c9-a471-831d40610ceb" (UID: "69cd086b-554c-49c9-a471-831d40610ceb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:14.546466 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.546299    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-workload-certs" pod "69cd086b-554c-49c9-a471-831d40610ceb" (UID: "69cd086b-554c-49c9-a471-831d40610ceb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:14.546550 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.546529    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69cd086b-554c-49c9-a471-831d40610ceb-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:14.546609 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.546557    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:14.546609 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.546573    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:14.546609 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.546587    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:14.546609 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.546602    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:14.548105 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.548086    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69cd086b-554c-49c9-a471-831d40610ceb-istio-podinfo" pod "69cd086b-554c-49c9-a471-831d40610ceb" (UID: "69cd086b-554c-49c9-a471-831d40610ceb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:50:14.548178 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.548138    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69cd086b-554c-49c9-a471-831d40610ceb-kube-api-access-g5tng" pod "69cd086b-554c-49c9-a471-831d40610ceb" (UID: "69cd086b-554c-49c9-a471-831d40610ceb"). InnerVolumeSpecName "kube-api-access-g5tng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:50:14.548277 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.548256    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69cd086b-554c-49c9-a471-831d40610ceb-istio-token" pod "69cd086b-554c-49c9-a471-831d40610ceb" (UID: "69cd086b-554c-49c9-a471-831d40610ceb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:50:14.548333 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.548261    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-istio-envoy" pod "69cd086b-554c-49c9-a471-831d40610ceb" (UID: "69cd086b-554c-49c9-a471-831d40610ceb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:50:14.647899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.647862    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69cd086b-554c-49c9-a471-831d40610ceb-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:14.647899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.647891    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5tng\" (UniqueName: \"kubernetes.io/projected/69cd086b-554c-49c9-a471-831d40610ceb-kube-api-access-g5tng\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:14.647899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.647903    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69cd086b-554c-49c9-a471-831d40610ceb-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:14.648124 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:14.647912    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69cd086b-554c-49c9-a471-831d40610ceb-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:50:15.331108 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:15.331025    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"
Nov 01 06:50:15.331108 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:15.331064    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k" event={"ID":"69cd086b-554c-49c9-a471-831d40610ceb","Type":"ContainerDied","Data":"68ee04acb249ee0a21c51b58706098df7df234407a311421227a3198a7b1002a"}
Nov 01 06:50:15.331625 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:15.331140    5619 scope.go:122] "RemoveContainer" containerID="a135903a60c7cfdd5993c3726afbf831edd114bcd689f1f19b7971081968b924"
Nov 01 06:50:15.549554 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:15.549520    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"]
Nov 01 06:50:15.600207 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:15.600176    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-76s2k"]
Nov 01 06:50:16.270813 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:16.270781    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:16.961587 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:16.961552    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69cd086b-554c-49c9-a471-831d40610ceb" path="/var/lib/kubelet/pods/69cd086b-554c-49c9-a471-831d40610ceb/volumes"
Nov 01 06:50:21.066509 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:21.066470    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:25.867166 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:25.867115    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:25.880423 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:50:25.880393    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:50:30.770709 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:30.770671    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:34.047537 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:34.047492    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:35.670436 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:35.670396    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:38.457636 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:38.457598    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:40.361104 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:40.361070    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:41.588711 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:41.588676    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:41.736074 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:41.736042    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:41.882658 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:41.882621    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:44.091722 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:44.091682    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:44.942767 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:44.942727    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:45.124313 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:45.124144    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:49.868057 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:49.868016    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:52.569939 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:52.569896    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:52.770210 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:52.770177    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:54.674570 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:54.674524    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:54.733594 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:54.733564    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"]
Nov 01 06:50:54.823677 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:54.823600    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:54.974125 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:54.974028    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:55.403754 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:55.403717    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:55.879451 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:50:55.879415    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:50:59.379526 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:59.379491    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:50:59.734252 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:59.734118    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r" podUID="4cc4e41a-b8cb-4d9c-a1a6-07c701659943" containerName="payload-processing" containerID="cri-o://aca767221d3954981a53f3b2cfcf09c394b8a7dd9235df9f5bddbb591c63dfb3" gracePeriod=25
Nov 01 06:50:59.799346 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:50:59.799311    5619 logging.go:55] [core] [Channel #6988 SubChannel #6989]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.156:50052", ServerName: "10.133.0.156:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.156:50052: connect: connection refused"
Nov 01 06:50:59.988284 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:50:59.988223    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"
Nov 01 06:51:00.015988 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.015961    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-provider-credentials\") pod \"4cc4e41a-b8cb-4d9c-a1a6-07c701659943\" (UID: \"4cc4e41a-b8cb-4d9c-a1a6-07c701659943\") "
Nov 01 06:51:00.016136 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.015993    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-plugins-config-volume\") pod \"4cc4e41a-b8cb-4d9c-a1a6-07c701659943\" (UID: \"4cc4e41a-b8cb-4d9c-a1a6-07c701659943\") "
Nov 01 06:51:00.016136 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.016032    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-kube-api-access-627jz\" (UniqueName: \"kubernetes.io/projected/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-kube-api-access-627jz\") pod \"4cc4e41a-b8cb-4d9c-a1a6-07c701659943\" (UID: \"4cc4e41a-b8cb-4d9c-a1a6-07c701659943\") "
Nov 01 06:51:00.016272 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.016205    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-routing-overlay-volume\") pod \"4cc4e41a-b8cb-4d9c-a1a6-07c701659943\" (UID: \"4cc4e41a-b8cb-4d9c-a1a6-07c701659943\") "
Nov 01 06:51:00.016548 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.016514    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-plugins-config-volume" pod "4cc4e41a-b8cb-4d9c-a1a6-07c701659943" (UID: "4cc4e41a-b8cb-4d9c-a1a6-07c701659943"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:51:00.016656 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.016590    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-routing-overlay-volume" pod "4cc4e41a-b8cb-4d9c-a1a6-07c701659943" (UID: "4cc4e41a-b8cb-4d9c-a1a6-07c701659943"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:51:00.016773 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.016755    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:00.016810 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.016796    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:00.018121 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.018101    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-provider-credentials" pod "4cc4e41a-b8cb-4d9c-a1a6-07c701659943" (UID: "4cc4e41a-b8cb-4d9c-a1a6-07c701659943"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:51:00.018203 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.018187    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-kube-api-access-627jz" pod "4cc4e41a-b8cb-4d9c-a1a6-07c701659943" (UID: "4cc4e41a-b8cb-4d9c-a1a6-07c701659943"). InnerVolumeSpecName "kube-api-access-627jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:51:00.118073 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.118045    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:00.118073 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.118072    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-627jz\" (UniqueName: \"kubernetes.io/projected/4cc4e41a-b8cb-4d9c-a1a6-07c701659943-kube-api-access-627jz\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:00.593181 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.593142    5619 generic.go:356] "Generic (PLEG): container finished" podID="4cc4e41a-b8cb-4d9c-a1a6-07c701659943" containerID="aca767221d3954981a53f3b2cfcf09c394b8a7dd9235df9f5bddbb591c63dfb3" exitCode=0
Nov 01 06:51:00.593659 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.593191    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"
Nov 01 06:51:00.593659 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.593223    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r" event={"ID":"4cc4e41a-b8cb-4d9c-a1a6-07c701659943","Type":"ContainerDied","Data":"aca767221d3954981a53f3b2cfcf09c394b8a7dd9235df9f5bddbb591c63dfb3"}
Nov 01 06:51:00.593659 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.593279    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r" event={"ID":"4cc4e41a-b8cb-4d9c-a1a6-07c701659943","Type":"ContainerDied","Data":"e94a6ec3a5a690596949047fa6e1875b1c47d943eeb6ccbd029273e6c6552ff0"}
Nov 01 06:51:00.593659 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.593303    5619 scope.go:122] "RemoveContainer" containerID="aca767221d3954981a53f3b2cfcf09c394b8a7dd9235df9f5bddbb591c63dfb3"
Nov 01 06:51:00.603145 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.603127    5619 scope.go:122] "RemoveContainer" containerID="aca767221d3954981a53f3b2cfcf09c394b8a7dd9235df9f5bddbb591c63dfb3"
Nov 01 06:51:00.603454 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:51:00.603433    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aca767221d3954981a53f3b2cfcf09c394b8a7dd9235df9f5bddbb591c63dfb3\": container with ID starting with aca767221d3954981a53f3b2cfcf09c394b8a7dd9235df9f5bddbb591c63dfb3 not found: ID does not exist" containerID="aca767221d3954981a53f3b2cfcf09c394b8a7dd9235df9f5bddbb591c63dfb3"
Nov 01 06:51:00.603508 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.603464    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aca767221d3954981a53f3b2cfcf09c394b8a7dd9235df9f5bddbb591c63dfb3"} err="failed to get container status \"aca767221d3954981a53f3b2cfcf09c394b8a7dd9235df9f5bddbb591c63dfb3\": rpc error: code = NotFound desc = could not find container \"aca767221d3954981a53f3b2cfcf09c394b8a7dd9235df9f5bddbb591c63dfb3\": container with ID starting with aca767221d3954981a53f3b2cfcf09c394b8a7dd9235df9f5bddbb591c63dfb3 not found: ID does not exist"
Nov 01 06:51:00.761573 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.761538    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"]
Nov 01 06:51:00.799883 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.799841    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r" podUID="4cc4e41a-b8cb-4d9c-a1a6-07c701659943" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.156:50052\" within 1s: context deadline exceeded"
Nov 01 06:51:00.799883 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:51:00.799877    5619 logging.go:55] [core] [Channel #6988 SubChannel #6989]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.156:50052", ServerName: "10.133.0.156:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.156:50052: operation was canceled"
Nov 01 06:51:00.812620 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.812589    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-5be4fa/payload-processing-external-model-e2e-extmod-a-5be4fa-6d76n449r"]
Nov 01 06:51:00.962177 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:00.962084    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cc4e41a-b8cb-4d9c-a1a6-07c701659943" path="/var/lib/kubelet/pods/4cc4e41a-b8cb-4d9c-a1a6-07c701659943/volumes"
Nov 01 06:51:01.505103 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:01.505059    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:01.714026 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:01.713992    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:01.952679 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:01.952639    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:03.558912 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:03.558857    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:04.173967 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:04.173931    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:04.900846 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:04.900798    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:05.358410 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:05.358371    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:08.375466 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:08.375430    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:08.982314 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:08.982275    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:09.781160 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:09.781126    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns"]
Nov 01 06:51:09.888709 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:09.888679    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc"]
Nov 01 06:51:10.145299 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:10.145262    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:10.293607 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:10.293568    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:10.494395 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:10.494291    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:13.867944 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:13.867904    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:14.287446 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:14.287316    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:14.781891 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:14.781829    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns" podUID="22cb8e7f-72a8-4cf7-a419-8470c27d8799" containerName="payload-processing" containerID="cri-o://9b0e1c40dd88bc5942d60a95e52a2ca35f0f466a6d95f22f210868675a8e9c93" gracePeriod=25
Nov 01 06:51:14.889336 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:14.889287    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc" podUID="d3e57167-aba3-406a-9a21-a48ae4584959" containerName="payload-pre-processing" containerID="cri-o://4ccceef3d34c6cdaff340f6549499e84dffe02e8b654bc2477c7eff905d2c2a3" gracePeriod=25
Nov 01 06:51:18.578301 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:18.578262    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:19.933398 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:19.933355    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns" podUID="22cb8e7f-72a8-4cf7-a419-8470c27d8799" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:51:19.933773 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:19.933365    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc" podUID="d3e57167-aba3-406a-9a21-a48ae4584959" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:51:23.369829 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:23.369790    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:25.879668 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:51:25.879630    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:51:27.392222 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:27.392181    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"]
Nov 01 06:51:27.392635 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:27.392466    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr" podUID="c9089f8a-a6b6-4c53-b483-8b5991330c33" containerName="maas-api" containerID="cri-o://c32ff1c10c1842bd376508025682375bcd98123d2812e0bfaacedfc35f90583b" gracePeriod=30
Nov 01 06:51:27.547639 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:27.547608    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:28.391852 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.391808    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr" podUID="c9089f8a-a6b6-4c53-b483-8b5991330c33" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.151:8443/health\": dial tcp 10.133.0.151:8443: connect: connection refused"
Nov 01 06:51:28.657600 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.657539    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"
Nov 01 06:51:28.725086 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.725054    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9089f8a-a6b6-4c53-b483-8b5991330c33-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9089f8a-a6b6-4c53-b483-8b5991330c33-maas-api-tls\") pod \"c9089f8a-a6b6-4c53-b483-8b5991330c33\" (UID: \"c9089f8a-a6b6-4c53-b483-8b5991330c33\") "
Nov 01 06:51:28.725242 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.725167    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9089f8a-a6b6-4c53-b483-8b5991330c33-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9089f8a-a6b6-4c53-b483-8b5991330c33-openshift-service-ca\") pod \"c9089f8a-a6b6-4c53-b483-8b5991330c33\" (UID: \"c9089f8a-a6b6-4c53-b483-8b5991330c33\") "
Nov 01 06:51:28.725242 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.725196    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9089f8a-a6b6-4c53-b483-8b5991330c33-kube-api-access-gfckb\" (UniqueName: \"kubernetes.io/projected/c9089f8a-a6b6-4c53-b483-8b5991330c33-kube-api-access-gfckb\") pod \"c9089f8a-a6b6-4c53-b483-8b5991330c33\" (UID: \"c9089f8a-a6b6-4c53-b483-8b5991330c33\") "
Nov 01 06:51:28.725242 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.725235    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9089f8a-a6b6-4c53-b483-8b5991330c33-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9089f8a-a6b6-4c53-b483-8b5991330c33-metrics-certs\") pod \"c9089f8a-a6b6-4c53-b483-8b5991330c33\" (UID: \"c9089f8a-a6b6-4c53-b483-8b5991330c33\") "
Nov 01 06:51:28.725602 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.725573    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9089f8a-a6b6-4c53-b483-8b5991330c33-openshift-service-ca" pod "c9089f8a-a6b6-4c53-b483-8b5991330c33" (UID: "c9089f8a-a6b6-4c53-b483-8b5991330c33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:51:28.727169 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.727143    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9089f8a-a6b6-4c53-b483-8b5991330c33-maas-api-tls" pod "c9089f8a-a6b6-4c53-b483-8b5991330c33" (UID: "c9089f8a-a6b6-4c53-b483-8b5991330c33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:51:28.727297 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.727253    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9089f8a-a6b6-4c53-b483-8b5991330c33-metrics-certs" pod "c9089f8a-a6b6-4c53-b483-8b5991330c33" (UID: "c9089f8a-a6b6-4c53-b483-8b5991330c33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:51:28.727406 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.727386    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9089f8a-a6b6-4c53-b483-8b5991330c33-kube-api-access-gfckb" pod "c9089f8a-a6b6-4c53-b483-8b5991330c33" (UID: "c9089f8a-a6b6-4c53-b483-8b5991330c33"). InnerVolumeSpecName "kube-api-access-gfckb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:51:28.759044 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.759012    5619 generic.go:356] "Generic (PLEG): container finished" podID="c9089f8a-a6b6-4c53-b483-8b5991330c33" containerID="c32ff1c10c1842bd376508025682375bcd98123d2812e0bfaacedfc35f90583b" exitCode=0
Nov 01 06:51:28.759184 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.759056    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr" event={"ID":"c9089f8a-a6b6-4c53-b483-8b5991330c33","Type":"ContainerDied","Data":"c32ff1c10c1842bd376508025682375bcd98123d2812e0bfaacedfc35f90583b"}
Nov 01 06:51:28.759184 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.759112    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr" event={"ID":"c9089f8a-a6b6-4c53-b483-8b5991330c33","Type":"ContainerDied","Data":"9c28dd433ffbc85fb6cb8a455833531b836157038149f7683d423b6deb1e17ca"}
Nov 01 06:51:28.759184 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.759137    5619 scope.go:122] "RemoveContainer" containerID="c32ff1c10c1842bd376508025682375bcd98123d2812e0bfaacedfc35f90583b"
Nov 01 06:51:28.759184 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.759069    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"
Nov 01 06:51:28.770384 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.770199    5619 scope.go:122] "RemoveContainer" containerID="cdd9cd9b9dce913f7b01cf4381146db3f641f8cd556ce25a40f8a6f05247189b"
Nov 01 06:51:28.780173 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.780153    5619 scope.go:122] "RemoveContainer" containerID="c32ff1c10c1842bd376508025682375bcd98123d2812e0bfaacedfc35f90583b"
Nov 01 06:51:28.780511 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:51:28.780483    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c32ff1c10c1842bd376508025682375bcd98123d2812e0bfaacedfc35f90583b\": container with ID starting with c32ff1c10c1842bd376508025682375bcd98123d2812e0bfaacedfc35f90583b not found: ID does not exist" containerID="c32ff1c10c1842bd376508025682375bcd98123d2812e0bfaacedfc35f90583b"
Nov 01 06:51:28.780610 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.780518    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c32ff1c10c1842bd376508025682375bcd98123d2812e0bfaacedfc35f90583b"} err="failed to get container status \"c32ff1c10c1842bd376508025682375bcd98123d2812e0bfaacedfc35f90583b\": rpc error: code = NotFound desc = could not find container \"c32ff1c10c1842bd376508025682375bcd98123d2812e0bfaacedfc35f90583b\": container with ID starting with c32ff1c10c1842bd376508025682375bcd98123d2812e0bfaacedfc35f90583b not found: ID does not exist"
Nov 01 06:51:28.780610 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.780542    5619 scope.go:122] "RemoveContainer" containerID="cdd9cd9b9dce913f7b01cf4381146db3f641f8cd556ce25a40f8a6f05247189b"
Nov 01 06:51:28.780852 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:51:28.780828    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd9cd9b9dce913f7b01cf4381146db3f641f8cd556ce25a40f8a6f05247189b\": container with ID starting with cdd9cd9b9dce913f7b01cf4381146db3f641f8cd556ce25a40f8a6f05247189b not found: ID does not exist" containerID="cdd9cd9b9dce913f7b01cf4381146db3f641f8cd556ce25a40f8a6f05247189b"
Nov 01 06:51:28.780985 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.780858    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd9cd9b9dce913f7b01cf4381146db3f641f8cd556ce25a40f8a6f05247189b"} err="failed to get container status \"cdd9cd9b9dce913f7b01cf4381146db3f641f8cd556ce25a40f8a6f05247189b\": rpc error: code = NotFound desc = could not find container \"cdd9cd9b9dce913f7b01cf4381146db3f641f8cd556ce25a40f8a6f05247189b\": container with ID starting with cdd9cd9b9dce913f7b01cf4381146db3f641f8cd556ce25a40f8a6f05247189b not found: ID does not exist"
Nov 01 06:51:28.826937 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.826910    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9089f8a-a6b6-4c53-b483-8b5991330c33-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:28.826937 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.826934    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9089f8a-a6b6-4c53-b483-8b5991330c33-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:28.827132 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.826948    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9089f8a-a6b6-4c53-b483-8b5991330c33-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:28.827132 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.826958    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfckb\" (UniqueName: \"kubernetes.io/projected/c9089f8a-a6b6-4c53-b483-8b5991330c33-kube-api-access-gfckb\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:28.892391 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.892356    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"]
Nov 01 06:51:28.896290 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.896266    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5be4fa-f774b886c-zlgmr"]
Nov 01 06:51:28.961715 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:28.961634    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9089f8a-a6b6-4c53-b483-8b5991330c33" path="/var/lib/kubelet/pods/c9089f8a-a6b6-4c53-b483-8b5991330c33/volumes"
Nov 01 06:51:29.283667 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:29.283579    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:29.429612 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:29.429575    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:29.933607 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:29.933569    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc" podUID="d3e57167-aba3-406a-9a21-a48ae4584959" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:51:29.934001 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:29.933569    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns" podUID="22cb8e7f-72a8-4cf7-a419-8470c27d8799" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:51:30.942187 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:30.942147    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:31.477291 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:31.477258    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:32.190497 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:32.190457    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:34.371403 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:34.371362    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:34.520241 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:34.520203    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:35.048356 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.048274    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns"
Nov 01 06:51:35.097358 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.095193    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22cb8e7f-72a8-4cf7-a419-8470c27d8799-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22cb8e7f-72a8-4cf7-a419-8470c27d8799-plugins-config-volume\") pod \"22cb8e7f-72a8-4cf7-a419-8470c27d8799\" (UID: \"22cb8e7f-72a8-4cf7-a419-8470c27d8799\") "
Nov 01 06:51:35.097358 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.095315    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22cb8e7f-72a8-4cf7-a419-8470c27d8799-kube-api-access-d4wdr\" (UniqueName: \"kubernetes.io/projected/22cb8e7f-72a8-4cf7-a419-8470c27d8799-kube-api-access-d4wdr\") pod \"22cb8e7f-72a8-4cf7-a419-8470c27d8799\" (UID: \"22cb8e7f-72a8-4cf7-a419-8470c27d8799\") "
Nov 01 06:51:35.097358 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.096888    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22cb8e7f-72a8-4cf7-a419-8470c27d8799-plugins-config-volume" pod "22cb8e7f-72a8-4cf7-a419-8470c27d8799" (UID: "22cb8e7f-72a8-4cf7-a419-8470c27d8799"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:51:35.105004 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.100741    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cb8e7f-72a8-4cf7-a419-8470c27d8799-kube-api-access-d4wdr" pod "22cb8e7f-72a8-4cf7-a419-8470c27d8799" (UID: "22cb8e7f-72a8-4cf7-a419-8470c27d8799"). InnerVolumeSpecName "kube-api-access-d4wdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:51:35.140220 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.140196    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc"
Nov 01 06:51:35.196118 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.196086    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3e57167-aba3-406a-9a21-a48ae4584959-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e57167-aba3-406a-9a21-a48ae4584959-plugins-config-volume\") pod \"d3e57167-aba3-406a-9a21-a48ae4584959\" (UID: \"d3e57167-aba3-406a-9a21-a48ae4584959\") "
Nov 01 06:51:35.196385 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.196355    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e57167-aba3-406a-9a21-a48ae4584959-kube-api-access-gnd2d\" (UniqueName: \"kubernetes.io/projected/d3e57167-aba3-406a-9a21-a48ae4584959-kube-api-access-gnd2d\") pod \"d3e57167-aba3-406a-9a21-a48ae4584959\" (UID: \"d3e57167-aba3-406a-9a21-a48ae4584959\") "
Nov 01 06:51:35.196551 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.196525    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e57167-aba3-406a-9a21-a48ae4584959-plugins-config-volume" pod "d3e57167-aba3-406a-9a21-a48ae4584959" (UID: "d3e57167-aba3-406a-9a21-a48ae4584959"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:51:35.196807 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.196783    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e57167-aba3-406a-9a21-a48ae4584959-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:35.196910 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.196809    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22cb8e7f-72a8-4cf7-a419-8470c27d8799-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:35.196910 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.196825    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4wdr\" (UniqueName: \"kubernetes.io/projected/22cb8e7f-72a8-4cf7-a419-8470c27d8799-kube-api-access-d4wdr\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:35.198944 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.198915    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e57167-aba3-406a-9a21-a48ae4584959-kube-api-access-gnd2d" pod "d3e57167-aba3-406a-9a21-a48ae4584959" (UID: "d3e57167-aba3-406a-9a21-a48ae4584959"). InnerVolumeSpecName "kube-api-access-gnd2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:51:35.297808 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.297766    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnd2d\" (UniqueName: \"kubernetes.io/projected/d3e57167-aba3-406a-9a21-a48ae4584959-kube-api-access-gnd2d\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:35.798971 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.798931    5619 generic.go:356] "Generic (PLEG): container finished" podID="d3e57167-aba3-406a-9a21-a48ae4584959" containerID="4ccceef3d34c6cdaff340f6549499e84dffe02e8b654bc2477c7eff905d2c2a3" exitCode=0
Nov 01 06:51:35.799435 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.798987    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc"
Nov 01 06:51:35.799435 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.799001    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc" event={"ID":"d3e57167-aba3-406a-9a21-a48ae4584959","Type":"ContainerDied","Data":"4ccceef3d34c6cdaff340f6549499e84dffe02e8b654bc2477c7eff905d2c2a3"}
Nov 01 06:51:35.799435 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.799048    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc" event={"ID":"d3e57167-aba3-406a-9a21-a48ae4584959","Type":"ContainerDied","Data":"4e87e00161b0bbcc506a4ead64895637af71e9947b42c3a7f18665cea339b126"}
Nov 01 06:51:35.799435 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.799073    5619 scope.go:122] "RemoveContainer" containerID="4ccceef3d34c6cdaff340f6549499e84dffe02e8b654bc2477c7eff905d2c2a3"
Nov 01 06:51:35.800824 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.800510    5619 generic.go:356] "Generic (PLEG): container finished" podID="22cb8e7f-72a8-4cf7-a419-8470c27d8799" containerID="9b0e1c40dd88bc5942d60a95e52a2ca35f0f466a6d95f22f210868675a8e9c93" exitCode=0
Nov 01 06:51:35.800824 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.800566    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns"
Nov 01 06:51:35.800824 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.800590    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns" event={"ID":"22cb8e7f-72a8-4cf7-a419-8470c27d8799","Type":"ContainerDied","Data":"9b0e1c40dd88bc5942d60a95e52a2ca35f0f466a6d95f22f210868675a8e9c93"}
Nov 01 06:51:35.800824 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.800628    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns" event={"ID":"22cb8e7f-72a8-4cf7-a419-8470c27d8799","Type":"ContainerDied","Data":"491e3c3b13386e8e1b430dceddf88873ec02db7d4bf2c9c46f3c6bc805972903"}
Nov 01 06:51:35.813583 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.813560    5619 scope.go:122] "RemoveContainer" containerID="4ccceef3d34c6cdaff340f6549499e84dffe02e8b654bc2477c7eff905d2c2a3"
Nov 01 06:51:35.813925 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:51:35.813896    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ccceef3d34c6cdaff340f6549499e84dffe02e8b654bc2477c7eff905d2c2a3\": container with ID starting with 4ccceef3d34c6cdaff340f6549499e84dffe02e8b654bc2477c7eff905d2c2a3 not found: ID does not exist" containerID="4ccceef3d34c6cdaff340f6549499e84dffe02e8b654bc2477c7eff905d2c2a3"
Nov 01 06:51:35.814022 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.813933    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ccceef3d34c6cdaff340f6549499e84dffe02e8b654bc2477c7eff905d2c2a3"} err="failed to get container status \"4ccceef3d34c6cdaff340f6549499e84dffe02e8b654bc2477c7eff905d2c2a3\": rpc error: code = NotFound desc = could not find container \"4ccceef3d34c6cdaff340f6549499e84dffe02e8b654bc2477c7eff905d2c2a3\": container with ID starting with 4ccceef3d34c6cdaff340f6549499e84dffe02e8b654bc2477c7eff905d2c2a3 not found: ID does not exist"
Nov 01 06:51:35.814022 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.813958    5619 scope.go:122] "RemoveContainer" containerID="9b0e1c40dd88bc5942d60a95e52a2ca35f0f466a6d95f22f210868675a8e9c93"
Nov 01 06:51:35.822864 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.822847    5619 scope.go:122] "RemoveContainer" containerID="9b0e1c40dd88bc5942d60a95e52a2ca35f0f466a6d95f22f210868675a8e9c93"
Nov 01 06:51:35.823186 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:51:35.823161    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b0e1c40dd88bc5942d60a95e52a2ca35f0f466a6d95f22f210868675a8e9c93\": container with ID starting with 9b0e1c40dd88bc5942d60a95e52a2ca35f0f466a6d95f22f210868675a8e9c93 not found: ID does not exist" containerID="9b0e1c40dd88bc5942d60a95e52a2ca35f0f466a6d95f22f210868675a8e9c93"
Nov 01 06:51:35.823285 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.823191    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b0e1c40dd88bc5942d60a95e52a2ca35f0f466a6d95f22f210868675a8e9c93"} err="failed to get container status \"9b0e1c40dd88bc5942d60a95e52a2ca35f0f466a6d95f22f210868675a8e9c93\": rpc error: code = NotFound desc = could not find container \"9b0e1c40dd88bc5942d60a95e52a2ca35f0f466a6d95f22f210868675a8e9c93\": container with ID starting with 9b0e1c40dd88bc5942d60a95e52a2ca35f0f466a6d95f22f210868675a8e9c93 not found: ID does not exist"
Nov 01 06:51:35.967271 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:35.967240    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc"]
Nov 01 06:51:36.017913 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:36.017872    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5be4fa-5c5c7c58f5-rbfjc"]
Nov 01 06:51:36.083315 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:36.083230    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns"]
Nov 01 06:51:36.088235 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:36.088205    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5be4fa-7bdb5cb75c-5hqns"]
Nov 01 06:51:36.961178 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:36.961145    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22cb8e7f-72a8-4cf7-a419-8470c27d8799" path="/var/lib/kubelet/pods/22cb8e7f-72a8-4cf7-a419-8470c27d8799/volumes"
Nov 01 06:51:36.961572 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:36.961542    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3e57167-aba3-406a-9a21-a48ae4584959" path="/var/lib/kubelet/pods/d3e57167-aba3-406a-9a21-a48ae4584959/volumes"
Nov 01 06:51:37.693792 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:37.693745    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:37.888294 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:37.888152    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:38.480254 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:38.480204    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:38.611128 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:38.611074    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"]
Nov 01 06:51:38.611368 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:38.611341    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx" podUID="21f94020-a328-405d-8604-81102600dbb2" containerName="maas-api" containerID="cri-o://4747ac46d16fae685ebacf42c779db5da9a2c5de5aaca71e88805ab106e1cd8a" gracePeriod=30
Nov 01 06:51:38.715942 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:38.715898    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:38.821285 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:38.821236    5619 generic.go:356] "Generic (PLEG): container finished" podID="21f94020-a328-405d-8604-81102600dbb2" containerID="4747ac46d16fae685ebacf42c779db5da9a2c5de5aaca71e88805ab106e1cd8a" exitCode=0
Nov 01 06:51:38.821495 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:38.821293    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx" event={"ID":"21f94020-a328-405d-8604-81102600dbb2","Type":"ContainerDied","Data":"4747ac46d16fae685ebacf42c779db5da9a2c5de5aaca71e88805ab106e1cd8a"}
Nov 01 06:51:38.859581 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:38.859549    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:38.879338 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:38.879301    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"
Nov 01 06:51:38.940551 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:38.940518    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21f94020-a328-405d-8604-81102600dbb2-kube-api-access-lxtdg\" (UniqueName: \"kubernetes.io/projected/21f94020-a328-405d-8604-81102600dbb2-kube-api-access-lxtdg\") pod \"21f94020-a328-405d-8604-81102600dbb2\" (UID: \"21f94020-a328-405d-8604-81102600dbb2\") "
Nov 01 06:51:38.940735 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:38.940580    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21f94020-a328-405d-8604-81102600dbb2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f94020-a328-405d-8604-81102600dbb2-maas-api-tls\") pod \"21f94020-a328-405d-8604-81102600dbb2\" (UID: \"21f94020-a328-405d-8604-81102600dbb2\") "
Nov 01 06:51:38.940735 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:38.940688    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21f94020-a328-405d-8604-81102600dbb2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f94020-a328-405d-8604-81102600dbb2-metrics-certs\") pod \"21f94020-a328-405d-8604-81102600dbb2\" (UID: \"21f94020-a328-405d-8604-81102600dbb2\") "
Nov 01 06:51:38.940866 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:38.940738    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21f94020-a328-405d-8604-81102600dbb2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f94020-a328-405d-8604-81102600dbb2-openshift-service-ca\") pod \"21f94020-a328-405d-8604-81102600dbb2\" (UID: \"21f94020-a328-405d-8604-81102600dbb2\") "
Nov 01 06:51:38.941150 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:38.941124    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f94020-a328-405d-8604-81102600dbb2-openshift-service-ca" pod "21f94020-a328-405d-8604-81102600dbb2" (UID: "21f94020-a328-405d-8604-81102600dbb2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:51:38.942791 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:38.942766    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21f94020-a328-405d-8604-81102600dbb2-maas-api-tls" pod "21f94020-a328-405d-8604-81102600dbb2" (UID: "21f94020-a328-405d-8604-81102600dbb2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:51:38.942882 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:38.942798    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f94020-a328-405d-8604-81102600dbb2-kube-api-access-lxtdg" pod "21f94020-a328-405d-8604-81102600dbb2" (UID: "21f94020-a328-405d-8604-81102600dbb2"). InnerVolumeSpecName "kube-api-access-lxtdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:51:38.942882 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:38.942826    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21f94020-a328-405d-8604-81102600dbb2-metrics-certs" pod "21f94020-a328-405d-8604-81102600dbb2" (UID: "21f94020-a328-405d-8604-81102600dbb2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:51:39.042673 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:39.042594    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f94020-a328-405d-8604-81102600dbb2-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:39.043108 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:39.043091    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f94020-a328-405d-8604-81102600dbb2-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:39.043149 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:39.043114    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxtdg\" (UniqueName: \"kubernetes.io/projected/21f94020-a328-405d-8604-81102600dbb2-kube-api-access-lxtdg\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:39.043149 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:39.043131    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21f94020-a328-405d-8604-81102600dbb2-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:39.827924 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:39.827887    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"
Nov 01 06:51:39.828526 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:39.827918    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx" event={"ID":"21f94020-a328-405d-8604-81102600dbb2","Type":"ContainerDied","Data":"fdc0de3d98ab394d49442b6dc6b3b14789e72d73c3b3ccdd19850b350e10f7f6"}
Nov 01 06:51:39.828526 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:39.827992    5619 scope.go:122] "RemoveContainer" containerID="4747ac46d16fae685ebacf42c779db5da9a2c5de5aaca71e88805ab106e1cd8a"
Nov 01 06:51:39.995880 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:39.995843    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"]
Nov 01 06:51:40.045510 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:40.045480    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d2260d-5bff9458f-ct4tx"]
Nov 01 06:51:40.647503 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:40.647468    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:40.962388 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:40.962276    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21f94020-a328-405d-8604-81102600dbb2" path="/var/lib/kubelet/pods/21f94020-a328-405d-8604-81102600dbb2/volumes"
Nov 01 06:51:44.328791 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:44.328750    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:45.461223 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:45.461186    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:47.925816 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:47.925778    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:48.549683 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:48.549648    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4"]
Nov 01 06:51:48.656421 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:48.656339    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9"]
Nov 01 06:51:48.904706 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:48.904677    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:49.090957 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:49.090920    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:50.315270 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:50.315232    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:53.533027 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:53.532986    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:53.550696 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:53.550638    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4" podUID="a8b15032-f506-448c-a9a6-f5f45930f3a6" containerName="payload-processing" containerID="cri-o://17bbfe362ebb797fa99877cc9cb4e0546ccc217ebc13d96a98e9a4435410750d" gracePeriod=25
Nov 01 06:51:53.657263 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:53.657223    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9" podUID="7f4f13d7-922b-4478-afe8-0b9c44d894ce" containerName="payload-pre-processing" containerID="cri-o://cdd885ad492c30ba322c7563606d8918fe81234355c04c56d8bfb547f8756c39" gracePeriod=25
Nov 01 06:51:54.248553 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.248515    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"]
Nov 01 06:51:54.248746 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.248726    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww" podUID="bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" containerName="istio-proxy" containerID="cri-o://e27fbac9595f220c71eab7004111fe0888f62db8ea53897b9b151dc0ed498cc3" gracePeriod=30
Nov 01 06:51:54.423573 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.423541    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"]
Nov 01 06:51:54.423768 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.423745    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g" podUID="1b1d6695-9e95-46fe-b1c4-f6f4b7573f67" containerName="maas-api" containerID="cri-o://4ca161de389a15eb2f6f89991120695d8a69a0cea0e6ab5eaba172951b0e329e" gracePeriod=30
Nov 01 06:51:54.487735 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.487703    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:54.700730 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.700705    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"
Nov 01 06:51:54.728041 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.728012    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-metrics-certs\") pod \"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67\" (UID: \"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67\") "
Nov 01 06:51:54.728169 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.728052    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-maas-api-tls\") pod \"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67\" (UID: \"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67\") "
Nov 01 06:51:54.728169 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.728073    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-openshift-service-ca\") pod \"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67\" (UID: \"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67\") "
Nov 01 06:51:54.728169 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.728107    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-kube-api-access-pln2k\" (UniqueName: \"kubernetes.io/projected/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-kube-api-access-pln2k\") pod \"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67\" (UID: \"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67\") "
Nov 01 06:51:54.728518 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.728487    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-openshift-service-ca" pod "1b1d6695-9e95-46fe-b1c4-f6f4b7573f67" (UID: "1b1d6695-9e95-46fe-b1c4-f6f4b7573f67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:51:54.728658 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.728640    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:54.730077 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.730051    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-metrics-certs" pod "1b1d6695-9e95-46fe-b1c4-f6f4b7573f67" (UID: "1b1d6695-9e95-46fe-b1c4-f6f4b7573f67"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:51:54.730185 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.730098    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-maas-api-tls" pod "1b1d6695-9e95-46fe-b1c4-f6f4b7573f67" (UID: "1b1d6695-9e95-46fe-b1c4-f6f4b7573f67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:51:54.730185 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.730156    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-kube-api-access-pln2k" pod "1b1d6695-9e95-46fe-b1c4-f6f4b7573f67" (UID: "1b1d6695-9e95-46fe-b1c4-f6f4b7573f67"). InnerVolumeSpecName "kube-api-access-pln2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:51:54.829788 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.829708    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:54.829788 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.829745    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:54.829788 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.829759    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pln2k\" (UniqueName: \"kubernetes.io/projected/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67-kube-api-access-pln2k\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:54.911529 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.911496    5619 generic.go:356] "Generic (PLEG): container finished" podID="1b1d6695-9e95-46fe-b1c4-f6f4b7573f67" containerID="4ca161de389a15eb2f6f89991120695d8a69a0cea0e6ab5eaba172951b0e329e" exitCode=0
Nov 01 06:51:54.911736 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.911548    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"
Nov 01 06:51:54.911736 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.911585    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g" event={"ID":"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67","Type":"ContainerDied","Data":"4ca161de389a15eb2f6f89991120695d8a69a0cea0e6ab5eaba172951b0e329e"}
Nov 01 06:51:54.911736 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.911644    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g" event={"ID":"1b1d6695-9e95-46fe-b1c4-f6f4b7573f67","Type":"ContainerDied","Data":"63a8bcb9afd314b131b33484a356494ef072264ee5e65bc8a4f6f036c334d03a"}
Nov 01 06:51:54.911736 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.911674    5619 scope.go:122] "RemoveContainer" containerID="4ca161de389a15eb2f6f89991120695d8a69a0cea0e6ab5eaba172951b0e329e"
Nov 01 06:51:54.921447 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.921431    5619 scope.go:122] "RemoveContainer" containerID="4ca161de389a15eb2f6f89991120695d8a69a0cea0e6ab5eaba172951b0e329e"
Nov 01 06:51:54.921695 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:51:54.921678    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ca161de389a15eb2f6f89991120695d8a69a0cea0e6ab5eaba172951b0e329e\": container with ID starting with 4ca161de389a15eb2f6f89991120695d8a69a0cea0e6ab5eaba172951b0e329e not found: ID does not exist" containerID="4ca161de389a15eb2f6f89991120695d8a69a0cea0e6ab5eaba172951b0e329e"
Nov 01 06:51:54.921753 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:54.921703    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca161de389a15eb2f6f89991120695d8a69a0cea0e6ab5eaba172951b0e329e"} err="failed to get container status \"4ca161de389a15eb2f6f89991120695d8a69a0cea0e6ab5eaba172951b0e329e\": rpc error: code = NotFound desc = could not find container \"4ca161de389a15eb2f6f89991120695d8a69a0cea0e6ab5eaba172951b0e329e\": container with ID starting with 4ca161de389a15eb2f6f89991120695d8a69a0cea0e6ab5eaba172951b0e329e not found: ID does not exist"
Nov 01 06:51:55.079023 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:55.078990    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"]
Nov 01 06:51:55.083383 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:55.083303    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d7d444-77cb89c77b-5nt4g"]
Nov 01 06:51:55.879755 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:51:55.879720    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:51:56.259170 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:56.259083    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:56.962830 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:56.962792    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b1d6695-9e95-46fe-b1c4-f6f4b7573f67" path="/var/lib/kubelet/pods/1b1d6695-9e95-46fe-b1c4-f6f4b7573f67/volumes"
Nov 01 06:51:59.312472 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.312436    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:59.500711 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.500676    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:51:59.507607 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.507582    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:51:59.579679 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.579612    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-workload-socket\") pod \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") "
Nov 01 06:51:59.579679 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.579660    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-podinfo\") pod \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") "
Nov 01 06:51:59.579904 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.579710    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-credential-socket\") pod \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") "
Nov 01 06:51:59.579904 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.579812    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-token\" (UniqueName: \"kubernetes.io/projected/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-token\") pod \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") "
Nov 01 06:51:59.579904 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.579845    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-envoy\") pod \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") "
Nov 01 06:51:59.579904 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.579879    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-data\") pod \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") "
Nov 01 06:51:59.579904 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.579882    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-workload-socket" pod "bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" (UID: "bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:51:59.580236 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.579975    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-workload-certs\") pod \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") "
Nov 01 06:51:59.580236 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.580016    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-credential-socket" pod "bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" (UID: "bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:51:59.580236 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.580047    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-kube-api-access-jg6dx\" (UniqueName: \"kubernetes.io/projected/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-kube-api-access-jg6dx\") pod \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") "
Nov 01 06:51:59.580236 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.580129    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-data" pod "bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" (UID: "bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:51:59.580236 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.580156    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istiod-ca-cert\") pod \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\" (UID: \"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea\") "
Nov 01 06:51:59.580547 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.580299    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-workload-certs" pod "bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" (UID: "bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:51:59.580638 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.580616    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:59.580700 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.580645    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:59.580700 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.580663    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:59.580700 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.580677    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:59.580700 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.580616    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istiod-ca-cert" pod "bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" (UID: "bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:51:59.582033 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.582004    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-podinfo" pod "bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" (UID: "bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:51:59.582408 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.582380    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-token" pod "bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" (UID: "bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:51:59.582497 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.582482    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-kube-api-access-jg6dx" pod "bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" (UID: "bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea"). InnerVolumeSpecName "kube-api-access-jg6dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:51:59.582543 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.582527    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-envoy" pod "bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" (UID: "bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:51:59.681815 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.681781    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:59.681815 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.681811    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:59.681815 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.681823    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg6dx\" (UniqueName: \"kubernetes.io/projected/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-kube-api-access-jg6dx\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:59.682038 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.681832    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:59.682038 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.681842    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:51:59.949984 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.949947    5619 generic.go:356] "Generic (PLEG): container finished" podID="bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" containerID="e27fbac9595f220c71eab7004111fe0888f62db8ea53897b9b151dc0ed498cc3" exitCode=0
Nov 01 06:51:59.950191 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.950003    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"
Nov 01 06:51:59.950191 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.950045    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww" event={"ID":"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea","Type":"ContainerDied","Data":"e27fbac9595f220c71eab7004111fe0888f62db8ea53897b9b151dc0ed498cc3"}
Nov 01 06:51:59.950191 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.950125    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww" event={"ID":"bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea","Type":"ContainerDied","Data":"79e96a96af2de3956d16c91442ebb2e401bf62c11f80ab05dd13e9d2371df474"}
Nov 01 06:51:59.950191 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.950159    5619 scope.go:122] "RemoveContainer" containerID="e27fbac9595f220c71eab7004111fe0888f62db8ea53897b9b151dc0ed498cc3"
Nov 01 06:51:59.962862 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.962841    5619 scope.go:122] "RemoveContainer" containerID="e27fbac9595f220c71eab7004111fe0888f62db8ea53897b9b151dc0ed498cc3"
Nov 01 06:51:59.963255 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:51:59.963229    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e27fbac9595f220c71eab7004111fe0888f62db8ea53897b9b151dc0ed498cc3\": container with ID starting with e27fbac9595f220c71eab7004111fe0888f62db8ea53897b9b151dc0ed498cc3 not found: ID does not exist" containerID="e27fbac9595f220c71eab7004111fe0888f62db8ea53897b9b151dc0ed498cc3"
Nov 01 06:51:59.963368 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:51:59.963262    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e27fbac9595f220c71eab7004111fe0888f62db8ea53897b9b151dc0ed498cc3"} err="failed to get container status \"e27fbac9595f220c71eab7004111fe0888f62db8ea53897b9b151dc0ed498cc3\": rpc error: code = NotFound desc = could not find container \"e27fbac9595f220c71eab7004111fe0888f62db8ea53897b9b151dc0ed498cc3\": container with ID starting with e27fbac9595f220c71eab7004111fe0888f62db8ea53897b9b151dc0ed498cc3 not found: ID does not exist"
Nov 01 06:52:00.117009 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:00.116977    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"]
Nov 01 06:52:00.168213 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:00.168172    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-8c4ww"]
Nov 01 06:52:00.917042 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:00.917005    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:00.963669 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:00.963626    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" path="/var/lib/kubelet/pods/bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea/volumes"
Nov 01 06:52:01.240524 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:01.240434    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:01.392488 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:01.392453    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9" podUID="7f4f13d7-922b-4478-afe8-0b9c44d894ce" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:52:01.392656 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:01.392533    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4" podUID="a8b15032-f506-448c-a9a6-f5f45930f3a6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:52:01.432705 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:01.432666    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:02.174004 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:02.173964    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:06.264749 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:06.264709    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:08.570198 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:08.570155    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:09.579857 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:09.579821    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c"]
Nov 01 06:52:09.685472 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:09.685428    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g"]
Nov 01 06:52:09.937688 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:09.937647    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:10.079852 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:10.079779    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:11.192211 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:11.192119    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:11.392273 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:11.392228    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9" podUID="7f4f13d7-922b-4478-afe8-0b9c44d894ce" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:52:11.392273 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:11.392250    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4" podUID="a8b15032-f506-448c-a9a6-f5f45930f3a6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:52:13.825042 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:13.824981    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4"
Nov 01 06:52:13.915601 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:13.915575    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9"
Nov 01 06:52:13.933487 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:13.933455    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b15032-f506-448c-a9a6-f5f45930f3a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b15032-f506-448c-a9a6-f5f45930f3a6-plugins-config-volume\") pod \"a8b15032-f506-448c-a9a6-f5f45930f3a6\" (UID: \"a8b15032-f506-448c-a9a6-f5f45930f3a6\") "
Nov 01 06:52:13.933657 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:13.933630    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b15032-f506-448c-a9a6-f5f45930f3a6-kube-api-access-hxdr5\" (UniqueName: \"kubernetes.io/projected/a8b15032-f506-448c-a9a6-f5f45930f3a6-kube-api-access-hxdr5\") pod \"a8b15032-f506-448c-a9a6-f5f45930f3a6\" (UID: \"a8b15032-f506-448c-a9a6-f5f45930f3a6\") "
Nov 01 06:52:13.933856 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:13.933831    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b15032-f506-448c-a9a6-f5f45930f3a6-plugins-config-volume" pod "a8b15032-f506-448c-a9a6-f5f45930f3a6" (UID: "a8b15032-f506-448c-a9a6-f5f45930f3a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:52:13.935641 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:13.935619    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b15032-f506-448c-a9a6-f5f45930f3a6-kube-api-access-hxdr5" pod "a8b15032-f506-448c-a9a6-f5f45930f3a6" (UID: "a8b15032-f506-448c-a9a6-f5f45930f3a6"). InnerVolumeSpecName "kube-api-access-hxdr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:52:14.027003 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.026967    5619 generic.go:356] "Generic (PLEG): container finished" podID="a8b15032-f506-448c-a9a6-f5f45930f3a6" containerID="17bbfe362ebb797fa99877cc9cb4e0546ccc217ebc13d96a98e9a4435410750d" exitCode=0
Nov 01 06:52:14.027195 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.027012    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4"
Nov 01 06:52:14.027195 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.027047    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4" event={"ID":"a8b15032-f506-448c-a9a6-f5f45930f3a6","Type":"ContainerDied","Data":"17bbfe362ebb797fa99877cc9cb4e0546ccc217ebc13d96a98e9a4435410750d"}
Nov 01 06:52:14.027195 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.027109    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4" event={"ID":"a8b15032-f506-448c-a9a6-f5f45930f3a6","Type":"ContainerDied","Data":"fed5680e37053de273e657833e20edfad08e8d99d8663e7c22711a309d3dccc8"}
Nov 01 06:52:14.027195 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.027145    5619 scope.go:122] "RemoveContainer" containerID="17bbfe362ebb797fa99877cc9cb4e0546ccc217ebc13d96a98e9a4435410750d"
Nov 01 06:52:14.028881 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.028825    5619 generic.go:356] "Generic (PLEG): container finished" podID="7f4f13d7-922b-4478-afe8-0b9c44d894ce" containerID="cdd885ad492c30ba322c7563606d8918fe81234355c04c56d8bfb547f8756c39" exitCode=0
Nov 01 06:52:14.028881 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.028871    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9" event={"ID":"7f4f13d7-922b-4478-afe8-0b9c44d894ce","Type":"ContainerDied","Data":"cdd885ad492c30ba322c7563606d8918fe81234355c04c56d8bfb547f8756c39"}
Nov 01 06:52:14.029011 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.028886    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9"
Nov 01 06:52:14.029011 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.028906    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9" event={"ID":"7f4f13d7-922b-4478-afe8-0b9c44d894ce","Type":"ContainerDied","Data":"1cdb0024846199dd915d3804f61c3ef2085c8199ae341cb98b284ca58e5179a2"}
Nov 01 06:52:14.034260 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.034233    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f4f13d7-922b-4478-afe8-0b9c44d894ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f4f13d7-922b-4478-afe8-0b9c44d894ce-plugins-config-volume\") pod \"7f4f13d7-922b-4478-afe8-0b9c44d894ce\" (UID: \"7f4f13d7-922b-4478-afe8-0b9c44d894ce\") "
Nov 01 06:52:14.034428 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.034349    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f4f13d7-922b-4478-afe8-0b9c44d894ce-kube-api-access-4s5zj\" (UniqueName: \"kubernetes.io/projected/7f4f13d7-922b-4478-afe8-0b9c44d894ce-kube-api-access-4s5zj\") pod \"7f4f13d7-922b-4478-afe8-0b9c44d894ce\" (UID: \"7f4f13d7-922b-4478-afe8-0b9c44d894ce\") "
Nov 01 06:52:14.034637 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.034617    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f4f13d7-922b-4478-afe8-0b9c44d894ce-plugins-config-volume" pod "7f4f13d7-922b-4478-afe8-0b9c44d894ce" (UID: "7f4f13d7-922b-4478-afe8-0b9c44d894ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:52:14.034923 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.034839    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f4f13d7-922b-4478-afe8-0b9c44d894ce-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:14.034923 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.034870    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxdr5\" (UniqueName: \"kubernetes.io/projected/a8b15032-f506-448c-a9a6-f5f45930f3a6-kube-api-access-hxdr5\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:14.034923 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.034888    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8b15032-f506-448c-a9a6-f5f45930f3a6-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:14.036574 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.036552    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4f13d7-922b-4478-afe8-0b9c44d894ce-kube-api-access-4s5zj" pod "7f4f13d7-922b-4478-afe8-0b9c44d894ce" (UID: "7f4f13d7-922b-4478-afe8-0b9c44d894ce"). InnerVolumeSpecName "kube-api-access-4s5zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:52:14.038167 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.038146    5619 scope.go:122] "RemoveContainer" containerID="17bbfe362ebb797fa99877cc9cb4e0546ccc217ebc13d96a98e9a4435410750d"
Nov 01 06:52:14.038500 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:52:14.038481    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17bbfe362ebb797fa99877cc9cb4e0546ccc217ebc13d96a98e9a4435410750d\": container with ID starting with 17bbfe362ebb797fa99877cc9cb4e0546ccc217ebc13d96a98e9a4435410750d not found: ID does not exist" containerID="17bbfe362ebb797fa99877cc9cb4e0546ccc217ebc13d96a98e9a4435410750d"
Nov 01 06:52:14.038563 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.038506    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17bbfe362ebb797fa99877cc9cb4e0546ccc217ebc13d96a98e9a4435410750d"} err="failed to get container status \"17bbfe362ebb797fa99877cc9cb4e0546ccc217ebc13d96a98e9a4435410750d\": rpc error: code = NotFound desc = could not find container \"17bbfe362ebb797fa99877cc9cb4e0546ccc217ebc13d96a98e9a4435410750d\": container with ID starting with 17bbfe362ebb797fa99877cc9cb4e0546ccc217ebc13d96a98e9a4435410750d not found: ID does not exist"
Nov 01 06:52:14.038563 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.038523    5619 scope.go:122] "RemoveContainer" containerID="cdd885ad492c30ba322c7563606d8918fe81234355c04c56d8bfb547f8756c39"
Nov 01 06:52:14.047261 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.047239    5619 scope.go:122] "RemoveContainer" containerID="cdd885ad492c30ba322c7563606d8918fe81234355c04c56d8bfb547f8756c39"
Nov 01 06:52:14.047614 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:52:14.047585    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd885ad492c30ba322c7563606d8918fe81234355c04c56d8bfb547f8756c39\": container with ID starting with cdd885ad492c30ba322c7563606d8918fe81234355c04c56d8bfb547f8756c39 not found: ID does not exist" containerID="cdd885ad492c30ba322c7563606d8918fe81234355c04c56d8bfb547f8756c39"
Nov 01 06:52:14.047681 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.047622    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd885ad492c30ba322c7563606d8918fe81234355c04c56d8bfb547f8756c39"} err="failed to get container status \"cdd885ad492c30ba322c7563606d8918fe81234355c04c56d8bfb547f8756c39\": rpc error: code = NotFound desc = could not find container \"cdd885ad492c30ba322c7563606d8918fe81234355c04c56d8bfb547f8756c39\": container with ID starting with cdd885ad492c30ba322c7563606d8918fe81234355c04c56d8bfb547f8756c39 not found: ID does not exist"
Nov 01 06:52:14.136290 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.136257    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s5zj\" (UniqueName: \"kubernetes.io/projected/7f4f13d7-922b-4478-afe8-0b9c44d894ce-kube-api-access-4s5zj\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:14.176272 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.176238    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4"]
Nov 01 06:52:14.180380 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.180341    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d7d444-5f96f88769-464w4"]
Nov 01 06:52:14.451585 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.451501    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9"]
Nov 01 06:52:14.457197 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.457171    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d7d444-6f67f7647-jckl9"]
Nov 01 06:52:14.530657 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.530620    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:14.580497 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.580458    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c" podUID="6b322967-2ec4-4195-b5dc-9524a46b5091" containerName="payload-processing" containerID="cri-o://71ed5d28285dd754bd1337cb2aacb71892965ce9c81dc1594207971fb52f8bb5" gracePeriod=25
Nov 01 06:52:14.685909 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.685865    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g" podUID="dbd66390-4712-4e35-acae-26c5b43e35ef" containerName="payload-pre-processing" containerID="cri-o://648fb08fd78fde93986e74bdf344d27f12b86928acae46c369c7d9f4a4f86458" gracePeriod=25
Nov 01 06:52:14.974062 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.974026    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f4f13d7-922b-4478-afe8-0b9c44d894ce" path="/var/lib/kubelet/pods/7f4f13d7-922b-4478-afe8-0b9c44d894ce/volumes"
Nov 01 06:52:14.974487 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:14.974458    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b15032-f506-448c-a9a6-f5f45930f3a6" path="/var/lib/kubelet/pods/a8b15032-f506-448c-a9a6-f5f45930f3a6/volumes"
Nov 01 06:52:17.162639 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:17.162599    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:18.371592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:18.371553    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c" podUID="6b322967-2ec4-4195-b5dc-9524a46b5091" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:52:18.371592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:18.371579    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g" podUID="dbd66390-4712-4e35-acae-26c5b43e35ef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:52:21.874440 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:21.874380    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:25.146969 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:25.146927    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:25.337429 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:25.337392    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:25.881947 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:52:25.881885    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:52:26.670953 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:26.670918    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:26.810679 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:26.810643    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:27.136533 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:27.136495    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:27.902281 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:27.902246    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:28.371505 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:28.371465    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g" podUID="dbd66390-4712-4e35-acae-26c5b43e35ef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:52:28.371678 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:28.371469    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c" podUID="6b322967-2ec4-4195-b5dc-9524a46b5091" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:52:31.767559 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:31.767523    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:33.124849 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:33.124813    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:33.319957 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:33.319921    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:34.159943 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:34.159904    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:34.742915 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:34.742870    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:34.842148 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:34.842103    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c"
Nov 01 06:52:34.936486 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:34.936456    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g"
Nov 01 06:52:34.947527 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:34.947508    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b322967-2ec4-4195-b5dc-9524a46b5091-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b322967-2ec4-4195-b5dc-9524a46b5091-plugins-config-volume\") pod \"6b322967-2ec4-4195-b5dc-9524a46b5091\" (UID: \"6b322967-2ec4-4195-b5dc-9524a46b5091\") "
Nov 01 06:52:34.947635 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:34.947619    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b322967-2ec4-4195-b5dc-9524a46b5091-kube-api-access-fdjsg\" (UniqueName: \"kubernetes.io/projected/6b322967-2ec4-4195-b5dc-9524a46b5091-kube-api-access-fdjsg\") pod \"6b322967-2ec4-4195-b5dc-9524a46b5091\" (UID: \"6b322967-2ec4-4195-b5dc-9524a46b5091\") "
Nov 01 06:52:34.947904 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:34.947880    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b322967-2ec4-4195-b5dc-9524a46b5091-plugins-config-volume" pod "6b322967-2ec4-4195-b5dc-9524a46b5091" (UID: "6b322967-2ec4-4195-b5dc-9524a46b5091"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:52:34.949511 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:34.949494    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b322967-2ec4-4195-b5dc-9524a46b5091-kube-api-access-fdjsg" pod "6b322967-2ec4-4195-b5dc-9524a46b5091" (UID: "6b322967-2ec4-4195-b5dc-9524a46b5091"). InnerVolumeSpecName "kube-api-access-fdjsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:52:35.048660 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.048633    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd66390-4712-4e35-acae-26c5b43e35ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd66390-4712-4e35-acae-26c5b43e35ef-plugins-config-volume\") pod \"dbd66390-4712-4e35-acae-26c5b43e35ef\" (UID: \"dbd66390-4712-4e35-acae-26c5b43e35ef\") "
Nov 01 06:52:35.048810 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.048668    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd66390-4712-4e35-acae-26c5b43e35ef-kube-api-access-d5xs4\" (UniqueName: \"kubernetes.io/projected/dbd66390-4712-4e35-acae-26c5b43e35ef-kube-api-access-d5xs4\") pod \"dbd66390-4712-4e35-acae-26c5b43e35ef\" (UID: \"dbd66390-4712-4e35-acae-26c5b43e35ef\") "
Nov 01 06:52:35.048909 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.048895    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b322967-2ec4-4195-b5dc-9524a46b5091-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:35.048949 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.048912    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdjsg\" (UniqueName: \"kubernetes.io/projected/6b322967-2ec4-4195-b5dc-9524a46b5091-kube-api-access-fdjsg\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:35.048996 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.048972    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd66390-4712-4e35-acae-26c5b43e35ef-plugins-config-volume" pod "dbd66390-4712-4e35-acae-26c5b43e35ef" (UID: "dbd66390-4712-4e35-acae-26c5b43e35ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:52:35.050692 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.050673    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd66390-4712-4e35-acae-26c5b43e35ef-kube-api-access-d5xs4" pod "dbd66390-4712-4e35-acae-26c5b43e35ef" (UID: "dbd66390-4712-4e35-acae-26c5b43e35ef"). InnerVolumeSpecName "kube-api-access-d5xs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:52:35.120137 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.120065    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:35.146683 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.146651    5619 generic.go:356] "Generic (PLEG): container finished" podID="6b322967-2ec4-4195-b5dc-9524a46b5091" containerID="71ed5d28285dd754bd1337cb2aacb71892965ce9c81dc1594207971fb52f8bb5" exitCode=0
Nov 01 06:52:35.146852 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.146697    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c"
Nov 01 06:52:35.146852 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.146743    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c" event={"ID":"6b322967-2ec4-4195-b5dc-9524a46b5091","Type":"ContainerDied","Data":"71ed5d28285dd754bd1337cb2aacb71892965ce9c81dc1594207971fb52f8bb5"}
Nov 01 06:52:35.146852 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.146797    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c" event={"ID":"6b322967-2ec4-4195-b5dc-9524a46b5091","Type":"ContainerDied","Data":"5e1687ec54551c9585ba996893e133516480e16452c52f8739c25b1b538857ba"}
Nov 01 06:52:35.146852 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.146828    5619 scope.go:122] "RemoveContainer" containerID="71ed5d28285dd754bd1337cb2aacb71892965ce9c81dc1594207971fb52f8bb5"
Nov 01 06:52:35.148287 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.148258    5619 generic.go:356] "Generic (PLEG): container finished" podID="dbd66390-4712-4e35-acae-26c5b43e35ef" containerID="648fb08fd78fde93986e74bdf344d27f12b86928acae46c369c7d9f4a4f86458" exitCode=0
Nov 01 06:52:35.148438 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.148299    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g"
Nov 01 06:52:35.148438 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.148352    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g" event={"ID":"dbd66390-4712-4e35-acae-26c5b43e35ef","Type":"ContainerDied","Data":"648fb08fd78fde93986e74bdf344d27f12b86928acae46c369c7d9f4a4f86458"}
Nov 01 06:52:35.148438 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.148390    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g" event={"ID":"dbd66390-4712-4e35-acae-26c5b43e35ef","Type":"ContainerDied","Data":"503dd00979cf6c9b2b71ab3f5bd0c06970e75c555f959e7fc6c6cfcbb308f002"}
Nov 01 06:52:35.149724 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.149697    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd66390-4712-4e35-acae-26c5b43e35ef-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:35.149724 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.149722    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5xs4\" (UniqueName: \"kubernetes.io/projected/dbd66390-4712-4e35-acae-26c5b43e35ef-kube-api-access-d5xs4\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:52:35.156855 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.156836    5619 scope.go:122] "RemoveContainer" containerID="71ed5d28285dd754bd1337cb2aacb71892965ce9c81dc1594207971fb52f8bb5"
Nov 01 06:52:35.157190 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:52:35.157162    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ed5d28285dd754bd1337cb2aacb71892965ce9c81dc1594207971fb52f8bb5\": container with ID starting with 71ed5d28285dd754bd1337cb2aacb71892965ce9c81dc1594207971fb52f8bb5 not found: ID does not exist" containerID="71ed5d28285dd754bd1337cb2aacb71892965ce9c81dc1594207971fb52f8bb5"
Nov 01 06:52:35.157302 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.157199    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ed5d28285dd754bd1337cb2aacb71892965ce9c81dc1594207971fb52f8bb5"} err="failed to get container status \"71ed5d28285dd754bd1337cb2aacb71892965ce9c81dc1594207971fb52f8bb5\": rpc error: code = NotFound desc = could not find container \"71ed5d28285dd754bd1337cb2aacb71892965ce9c81dc1594207971fb52f8bb5\": container with ID starting with 71ed5d28285dd754bd1337cb2aacb71892965ce9c81dc1594207971fb52f8bb5 not found: ID does not exist"
Nov 01 06:52:35.157302 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.157224    5619 scope.go:122] "RemoveContainer" containerID="648fb08fd78fde93986e74bdf344d27f12b86928acae46c369c7d9f4a4f86458"
Nov 01 06:52:35.165172 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.165157    5619 scope.go:122] "RemoveContainer" containerID="648fb08fd78fde93986e74bdf344d27f12b86928acae46c369c7d9f4a4f86458"
Nov 01 06:52:35.165450 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:52:35.165419    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"648fb08fd78fde93986e74bdf344d27f12b86928acae46c369c7d9f4a4f86458\": container with ID starting with 648fb08fd78fde93986e74bdf344d27f12b86928acae46c369c7d9f4a4f86458 not found: ID does not exist" containerID="648fb08fd78fde93986e74bdf344d27f12b86928acae46c369c7d9f4a4f86458"
Nov 01 06:52:35.165450 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.165439    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648fb08fd78fde93986e74bdf344d27f12b86928acae46c369c7d9f4a4f86458"} err="failed to get container status \"648fb08fd78fde93986e74bdf344d27f12b86928acae46c369c7d9f4a4f86458\": rpc error: code = NotFound desc = could not find container \"648fb08fd78fde93986e74bdf344d27f12b86928acae46c369c7d9f4a4f86458\": container with ID starting with 648fb08fd78fde93986e74bdf344d27f12b86928acae46c369c7d9f4a4f86458 not found: ID does not exist"
Nov 01 06:52:35.259621 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.259575    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:35.315033 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.314997    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g"]
Nov 01 06:52:35.319022 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.318996    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-747b7b-75bb67fc5-fqm2g"]
Nov 01 06:52:35.429594 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.429517    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c"]
Nov 01 06:52:35.433397 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:35.433371    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-747b7b-5d65d7fbb7-75x7c"]
Nov 01 06:52:36.013529 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:36.013493    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:36.962864 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:36.962834    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b322967-2ec4-4195-b5dc-9524a46b5091" path="/var/lib/kubelet/pods/6b322967-2ec4-4195-b5dc-9524a46b5091/volumes"
Nov 01 06:52:36.963254 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:36.963196    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd66390-4712-4e35-acae-26c5b43e35ef" path="/var/lib/kubelet/pods/dbd66390-4712-4e35-acae-26c5b43e35ef/volumes"
Nov 01 06:52:37.746675 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:37.746638    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:37.949239 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:37.949201    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:39.214420 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:39.214383    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:39.373081 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:39.373044    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:39.707089 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:39.707045    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:40.448923 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:40.448885    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:40.639271 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:40.639230    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:42.861310 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:42.861275    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:46.638681 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:46.638643    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:47.680637 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:47.680595    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:47.824739 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:47.824700    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:47.930806 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:47.930707    5619 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal" podUID="ef0f634a7979920e9df606f3bb474aac" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Nov 01 06:52:52.199817 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:52.199767    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:52.566112 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:52.566015    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:55.878568 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:52:55.878524    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:52:56.691786 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:56.691743    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:58.461771 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:58.461730    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:52:59.180714 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:52:59.180678    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:03.266568 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:03.266524    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:04.443413 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:04.443304    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:04.592449 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:04.592415    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:05.976231 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:05.976190    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:06.474917 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:06.474878    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:07.246215 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:07.246181    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:07.561183 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:07.561089    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:09.458549 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:09.458508    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:13.507225 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:13.507184    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:13.961773 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:13.961738    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6"]
Nov 01 06:53:14.068063 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:14.068029    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr"]
Nov 01 06:53:14.275404 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:14.274861    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:14.423619 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:14.423583    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:14.569819 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:14.569715    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:16.020966 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:16.020928    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:16.214951 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:16.214919    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:17.049841 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:17.049807    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:17.723806 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:17.723755    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:18.962078 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:18.962036    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6" podUID="7ec112e9-286e-42f5-be40-f6954033585c" containerName="payload-processing" containerID="cri-o://8528ac496f37e78c700a795f4c2e4b373e2fc0963ebaa2fb17f79307cd4af4c8" gracePeriod=25
Nov 01 06:53:19.069563 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:19.069521    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr" podUID="df8c3448-d616-414e-8e09-d39e82928731" containerName="payload-pre-processing" containerID="cri-o://ae6f3be15d6d0078921b00421a388ba4b2b28a394a747f4c9ddc07bd21e7a5db" gracePeriod=25
Nov 01 06:53:19.073865 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:19.073832    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:19.465647 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:19.465611    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:23.648848 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:23.648815    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"]
Nov 01 06:53:23.649255 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:23.649042    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj" podUID="1e3999df-3132-435d-93b4-0ed000180c4d" containerName="maas-api" containerID="cri-o://414ec24fa0b279e28804efe42b75f705899efc2850e86c2ac0863e5a8d9130d6" gracePeriod=30
Nov 01 06:53:23.751390 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:23.751351    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:23.925158 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:23.925131    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"
Nov 01 06:53:24.063019 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.062993    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e3999df-3132-435d-93b4-0ed000180c4d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e3999df-3132-435d-93b4-0ed000180c4d-metrics-certs\") pod \"1e3999df-3132-435d-93b4-0ed000180c4d\" (UID: \"1e3999df-3132-435d-93b4-0ed000180c4d\") "
Nov 01 06:53:24.063261 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.063237    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e3999df-3132-435d-93b4-0ed000180c4d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e3999df-3132-435d-93b4-0ed000180c4d-maas-api-tls\") pod \"1e3999df-3132-435d-93b4-0ed000180c4d\" (UID: \"1e3999df-3132-435d-93b4-0ed000180c4d\") "
Nov 01 06:53:24.063430 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.063302    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e3999df-3132-435d-93b4-0ed000180c4d-kube-api-access-59x6p\" (UniqueName: \"kubernetes.io/projected/1e3999df-3132-435d-93b4-0ed000180c4d-kube-api-access-59x6p\") pod \"1e3999df-3132-435d-93b4-0ed000180c4d\" (UID: \"1e3999df-3132-435d-93b4-0ed000180c4d\") "
Nov 01 06:53:24.063430 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.063420    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e3999df-3132-435d-93b4-0ed000180c4d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3999df-3132-435d-93b4-0ed000180c4d-openshift-service-ca\") pod \"1e3999df-3132-435d-93b4-0ed000180c4d\" (UID: \"1e3999df-3132-435d-93b4-0ed000180c4d\") "
Nov 01 06:53:24.064014 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.063988    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3999df-3132-435d-93b4-0ed000180c4d-openshift-service-ca" pod "1e3999df-3132-435d-93b4-0ed000180c4d" (UID: "1e3999df-3132-435d-93b4-0ed000180c4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:53:24.065704 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.065675    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3999df-3132-435d-93b4-0ed000180c4d-maas-api-tls" pod "1e3999df-3132-435d-93b4-0ed000180c4d" (UID: "1e3999df-3132-435d-93b4-0ed000180c4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:53:24.065920 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.065892    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3999df-3132-435d-93b4-0ed000180c4d-metrics-certs" pod "1e3999df-3132-435d-93b4-0ed000180c4d" (UID: "1e3999df-3132-435d-93b4-0ed000180c4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:53:24.066161 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.066128    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3999df-3132-435d-93b4-0ed000180c4d-kube-api-access-59x6p" pod "1e3999df-3132-435d-93b4-0ed000180c4d" (UID: "1e3999df-3132-435d-93b4-0ed000180c4d"). InnerVolumeSpecName "kube-api-access-59x6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:53:24.164858 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.164724    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59x6p\" (UniqueName: \"kubernetes.io/projected/1e3999df-3132-435d-93b4-0ed000180c4d-kube-api-access-59x6p\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:24.164858 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.164804    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3999df-3132-435d-93b4-0ed000180c4d-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:24.164858 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.164818    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e3999df-3132-435d-93b4-0ed000180c4d-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:24.164858 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.164832    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e3999df-3132-435d-93b4-0ed000180c4d-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:24.414541 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.414503    5619 generic.go:356] "Generic (PLEG): container finished" podID="1e3999df-3132-435d-93b4-0ed000180c4d" containerID="414ec24fa0b279e28804efe42b75f705899efc2850e86c2ac0863e5a8d9130d6" exitCode=0
Nov 01 06:53:24.414724 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.414556    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"
Nov 01 06:53:24.414724 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.414588    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj" event={"ID":"1e3999df-3132-435d-93b4-0ed000180c4d","Type":"ContainerDied","Data":"414ec24fa0b279e28804efe42b75f705899efc2850e86c2ac0863e5a8d9130d6"}
Nov 01 06:53:24.414724 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.414652    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj" event={"ID":"1e3999df-3132-435d-93b4-0ed000180c4d","Type":"ContainerDied","Data":"a1452b06cefd2bd8ff4078e8bd6cfa1cf7a5c644779f26e6ce593f35dc79927f"}
Nov 01 06:53:24.414724 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.414676    5619 scope.go:122] "RemoveContainer" containerID="414ec24fa0b279e28804efe42b75f705899efc2850e86c2ac0863e5a8d9130d6"
Nov 01 06:53:24.423995 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.423980    5619 scope.go:122] "RemoveContainer" containerID="00bc0c5b50df72d72aadaa78e1a251246d6a940fb3e54678207cbc0422614d01"
Nov 01 06:53:24.431635 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.431618    5619 scope.go:122] "RemoveContainer" containerID="414ec24fa0b279e28804efe42b75f705899efc2850e86c2ac0863e5a8d9130d6"
Nov 01 06:53:24.431904 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:53:24.431882    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"414ec24fa0b279e28804efe42b75f705899efc2850e86c2ac0863e5a8d9130d6\": container with ID starting with 414ec24fa0b279e28804efe42b75f705899efc2850e86c2ac0863e5a8d9130d6 not found: ID does not exist" containerID="414ec24fa0b279e28804efe42b75f705899efc2850e86c2ac0863e5a8d9130d6"
Nov 01 06:53:24.431957 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.431912    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414ec24fa0b279e28804efe42b75f705899efc2850e86c2ac0863e5a8d9130d6"} err="failed to get container status \"414ec24fa0b279e28804efe42b75f705899efc2850e86c2ac0863e5a8d9130d6\": rpc error: code = NotFound desc = could not find container \"414ec24fa0b279e28804efe42b75f705899efc2850e86c2ac0863e5a8d9130d6\": container with ID starting with 414ec24fa0b279e28804efe42b75f705899efc2850e86c2ac0863e5a8d9130d6 not found: ID does not exist"
Nov 01 06:53:24.431957 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.431930    5619 scope.go:122] "RemoveContainer" containerID="00bc0c5b50df72d72aadaa78e1a251246d6a940fb3e54678207cbc0422614d01"
Nov 01 06:53:24.432168 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:53:24.432148    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00bc0c5b50df72d72aadaa78e1a251246d6a940fb3e54678207cbc0422614d01\": container with ID starting with 00bc0c5b50df72d72aadaa78e1a251246d6a940fb3e54678207cbc0422614d01 not found: ID does not exist" containerID="00bc0c5b50df72d72aadaa78e1a251246d6a940fb3e54678207cbc0422614d01"
Nov 01 06:53:24.432227 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.432177    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00bc0c5b50df72d72aadaa78e1a251246d6a940fb3e54678207cbc0422614d01"} err="failed to get container status \"00bc0c5b50df72d72aadaa78e1a251246d6a940fb3e54678207cbc0422614d01\": rpc error: code = NotFound desc = could not find container \"00bc0c5b50df72d72aadaa78e1a251246d6a940fb3e54678207cbc0422614d01\": container with ID starting with 00bc0c5b50df72d72aadaa78e1a251246d6a940fb3e54678207cbc0422614d01 not found: ID does not exist"
Nov 01 06:53:24.584596 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.583028    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"]
Nov 01 06:53:24.612629 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.612590    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr" podUID="df8c3448-d616-414e-8e09-d39e82928731" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:53:24.612629 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.612616    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6" podUID="7ec112e9-286e-42f5-be40-f6954033585c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:53:24.632593 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.632560    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99e79c-5769b4d9c5-zlrfj"]
Nov 01 06:53:24.961242 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:24.961210    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e3999df-3132-435d-93b4-0ed000180c4d" path="/var/lib/kubelet/pods/1e3999df-3132-435d-93b4-0ed000180c4d/volumes"
Nov 01 06:53:25.472430 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:25.472392    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:25.879166 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:53:25.879135    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:53:28.259602 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:28.259562    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:29.084161 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:29.084129    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:30.254684 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:30.254635    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:32.807594 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:32.807553    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:32.954131 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:32.954082    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:33.964730 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:33.964687    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:34.113468 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:34.113432    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:34.612416 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:34.612377    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6" podUID="7ec112e9-286e-42f5-be40-f6954033585c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:53:34.612628 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:34.612381    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr" podUID="df8c3448-d616-414e-8e09-d39e82928731" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 06:53:35.092818 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:35.092768    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:35.292504 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:35.292469    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:35.496907 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:35.496820    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:35.932457 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:35.932394    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:35.978504 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:35.978476    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"]
Nov 01 06:53:35.978695 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:35.978674    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb" podUID="182e059b-08d3-4ea4-9a37-3bdc44581625" containerName="istio-proxy" containerID="cri-o://54e1ee34a189fd8676ef0e56cc458ef698f2f68e688776e17ccdfdb07fe39075" gracePeriod=30
Nov 01 06:53:36.083307 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:36.083273    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:36.544656 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:36.544624    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:36.913290 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:36.913260    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:37.109901 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:37.109863    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:38.597506 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:38.597474    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:38.743958 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:38.743918    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:39.234168 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.234089    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6"
Nov 01 06:53:39.309094 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.309065    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec112e9-286e-42f5-be40-f6954033585c-kube-api-access-rt47x\" (UniqueName: \"kubernetes.io/projected/7ec112e9-286e-42f5-be40-f6954033585c-kube-api-access-rt47x\") pod \"7ec112e9-286e-42f5-be40-f6954033585c\" (UID: \"7ec112e9-286e-42f5-be40-f6954033585c\") "
Nov 01 06:53:39.309243 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.309156    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ec112e9-286e-42f5-be40-f6954033585c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ec112e9-286e-42f5-be40-f6954033585c-plugins-config-volume\") pod \"7ec112e9-286e-42f5-be40-f6954033585c\" (UID: \"7ec112e9-286e-42f5-be40-f6954033585c\") "
Nov 01 06:53:39.309533 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.309510    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec112e9-286e-42f5-be40-f6954033585c-plugins-config-volume" pod "7ec112e9-286e-42f5-be40-f6954033585c" (UID: "7ec112e9-286e-42f5-be40-f6954033585c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:53:39.311144 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.311121    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec112e9-286e-42f5-be40-f6954033585c-kube-api-access-rt47x" pod "7ec112e9-286e-42f5-be40-f6954033585c" (UID: "7ec112e9-286e-42f5-be40-f6954033585c"). InnerVolumeSpecName "kube-api-access-rt47x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:53:39.327894 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.327865    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr"
Nov 01 06:53:39.410318 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.410282    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8c3448-d616-414e-8e09-d39e82928731-kube-api-access-drlsh\" (UniqueName: \"kubernetes.io/projected/df8c3448-d616-414e-8e09-d39e82928731-kube-api-access-drlsh\") pod \"df8c3448-d616-414e-8e09-d39e82928731\" (UID: \"df8c3448-d616-414e-8e09-d39e82928731\") "
Nov 01 06:53:39.410530 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.410403    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df8c3448-d616-414e-8e09-d39e82928731-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df8c3448-d616-414e-8e09-d39e82928731-plugins-config-volume\") pod \"df8c3448-d616-414e-8e09-d39e82928731\" (UID: \"df8c3448-d616-414e-8e09-d39e82928731\") "
Nov 01 06:53:39.410695 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.410677    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ec112e9-286e-42f5-be40-f6954033585c-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:39.410755 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.410701    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt47x\" (UniqueName: \"kubernetes.io/projected/7ec112e9-286e-42f5-be40-f6954033585c-kube-api-access-rt47x\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:39.410798 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.410755    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df8c3448-d616-414e-8e09-d39e82928731-plugins-config-volume" pod "df8c3448-d616-414e-8e09-d39e82928731" (UID: "df8c3448-d616-414e-8e09-d39e82928731"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:53:39.412562 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.412528    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8c3448-d616-414e-8e09-d39e82928731-kube-api-access-drlsh" pod "df8c3448-d616-414e-8e09-d39e82928731" (UID: "df8c3448-d616-414e-8e09-d39e82928731"). InnerVolumeSpecName "kube-api-access-drlsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:53:39.491851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.491762    5619 generic.go:356] "Generic (PLEG): container finished" podID="7ec112e9-286e-42f5-be40-f6954033585c" containerID="8528ac496f37e78c700a795f4c2e4b373e2fc0963ebaa2fb17f79307cd4af4c8" exitCode=0
Nov 01 06:53:39.491851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.491820    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6"
Nov 01 06:53:39.492071 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.491852    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6" event={"ID":"7ec112e9-286e-42f5-be40-f6954033585c","Type":"ContainerDied","Data":"8528ac496f37e78c700a795f4c2e4b373e2fc0963ebaa2fb17f79307cd4af4c8"}
Nov 01 06:53:39.492071 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.491905    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6" event={"ID":"7ec112e9-286e-42f5-be40-f6954033585c","Type":"ContainerDied","Data":"85634efbc0c861df86a58021a34ffbc5a728c768dbc685d334f646f81869cb9e"}
Nov 01 06:53:39.492071 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.491928    5619 scope.go:122] "RemoveContainer" containerID="8528ac496f37e78c700a795f4c2e4b373e2fc0963ebaa2fb17f79307cd4af4c8"
Nov 01 06:53:39.493481 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.493457    5619 generic.go:356] "Generic (PLEG): container finished" podID="df8c3448-d616-414e-8e09-d39e82928731" containerID="ae6f3be15d6d0078921b00421a388ba4b2b28a394a747f4c9ddc07bd21e7a5db" exitCode=0
Nov 01 06:53:39.493617 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.493526    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr"
Nov 01 06:53:39.493617 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.493577    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr" event={"ID":"df8c3448-d616-414e-8e09-d39e82928731","Type":"ContainerDied","Data":"ae6f3be15d6d0078921b00421a388ba4b2b28a394a747f4c9ddc07bd21e7a5db"}
Nov 01 06:53:39.493722 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.493634    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr" event={"ID":"df8c3448-d616-414e-8e09-d39e82928731","Type":"ContainerDied","Data":"9ed05961134c6306cddf49213930b26006a6f3c2a30c1a581570b9af174aeef1"}
Nov 01 06:53:39.504045 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.504022    5619 scope.go:122] "RemoveContainer" containerID="8528ac496f37e78c700a795f4c2e4b373e2fc0963ebaa2fb17f79307cd4af4c8"
Nov 01 06:53:39.504522 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:53:39.504393    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8528ac496f37e78c700a795f4c2e4b373e2fc0963ebaa2fb17f79307cd4af4c8\": container with ID starting with 8528ac496f37e78c700a795f4c2e4b373e2fc0963ebaa2fb17f79307cd4af4c8 not found: ID does not exist" containerID="8528ac496f37e78c700a795f4c2e4b373e2fc0963ebaa2fb17f79307cd4af4c8"
Nov 01 06:53:39.504996 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.504965    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8528ac496f37e78c700a795f4c2e4b373e2fc0963ebaa2fb17f79307cd4af4c8"} err="failed to get container status \"8528ac496f37e78c700a795f4c2e4b373e2fc0963ebaa2fb17f79307cd4af4c8\": rpc error: code = NotFound desc = could not find container \"8528ac496f37e78c700a795f4c2e4b373e2fc0963ebaa2fb17f79307cd4af4c8\": container with ID starting with 8528ac496f37e78c700a795f4c2e4b373e2fc0963ebaa2fb17f79307cd4af4c8 not found: ID does not exist"
Nov 01 06:53:39.504996 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.504997    5619 scope.go:122] "RemoveContainer" containerID="ae6f3be15d6d0078921b00421a388ba4b2b28a394a747f4c9ddc07bd21e7a5db"
Nov 01 06:53:39.505173 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.504922    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:39.512760 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.512737    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drlsh\" (UniqueName: \"kubernetes.io/projected/df8c3448-d616-414e-8e09-d39e82928731-kube-api-access-drlsh\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:39.512760 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.512759    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df8c3448-d616-414e-8e09-d39e82928731-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:39.516343 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.515704    5619 scope.go:122] "RemoveContainer" containerID="ae6f3be15d6d0078921b00421a388ba4b2b28a394a747f4c9ddc07bd21e7a5db"
Nov 01 06:53:39.517467 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:53:39.517441    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae6f3be15d6d0078921b00421a388ba4b2b28a394a747f4c9ddc07bd21e7a5db\": container with ID starting with ae6f3be15d6d0078921b00421a388ba4b2b28a394a747f4c9ddc07bd21e7a5db not found: ID does not exist" containerID="ae6f3be15d6d0078921b00421a388ba4b2b28a394a747f4c9ddc07bd21e7a5db"
Nov 01 06:53:39.517570 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.517474    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6f3be15d6d0078921b00421a388ba4b2b28a394a747f4c9ddc07bd21e7a5db"} err="failed to get container status \"ae6f3be15d6d0078921b00421a388ba4b2b28a394a747f4c9ddc07bd21e7a5db\": rpc error: code = NotFound desc = could not find container \"ae6f3be15d6d0078921b00421a388ba4b2b28a394a747f4c9ddc07bd21e7a5db\": container with ID starting with ae6f3be15d6d0078921b00421a388ba4b2b28a394a747f4c9ddc07bd21e7a5db not found: ID does not exist"
Nov 01 06:53:39.660646 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.660615    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6"]
Nov 01 06:53:39.664658 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.664631    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-99e79c-c97475487-wgjw6"]
Nov 01 06:53:39.774885 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.774801    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr"]
Nov 01 06:53:39.779150 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:39.779124    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99e79c-5549954495-8h8kr"]
Nov 01 06:53:40.368967 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:40.368928    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:40.960967 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:40.960923    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ec112e9-286e-42f5-be40-f6954033585c" path="/var/lib/kubelet/pods/7ec112e9-286e-42f5-be40-f6954033585c/volumes"
Nov 01 06:53:40.961524 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:40.961504    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df8c3448-d616-414e-8e09-d39e82928731" path="/var/lib/kubelet/pods/df8c3448-d616-414e-8e09-d39e82928731/volumes"
Nov 01 06:53:41.239156 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.239125    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:53:41.328407 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.328371    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-credential-socket\") pod \"182e059b-08d3-4ea4-9a37-3bdc44581625\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") "
Nov 01 06:53:41.328592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.328415    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-podinfo\") pod \"182e059b-08d3-4ea4-9a37-3bdc44581625\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") "
Nov 01 06:53:41.328592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.328478    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182e059b-08d3-4ea4-9a37-3bdc44581625-kube-api-access-z2hvt\" (UniqueName: \"kubernetes.io/projected/182e059b-08d3-4ea4-9a37-3bdc44581625-kube-api-access-z2hvt\") pod \"182e059b-08d3-4ea4-9a37-3bdc44581625\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") "
Nov 01 06:53:41.328592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.328511    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-workload-socket\") pod \"182e059b-08d3-4ea4-9a37-3bdc44581625\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") "
Nov 01 06:53:41.328592 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.328576    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-envoy\") pod \"182e059b-08d3-4ea4-9a37-3bdc44581625\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") "
Nov 01 06:53:41.328828 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.328638    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/182e059b-08d3-4ea4-9a37-3bdc44581625-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182e059b-08d3-4ea4-9a37-3bdc44581625-istiod-ca-cert\") pod \"182e059b-08d3-4ea4-9a37-3bdc44581625\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") "
Nov 01 06:53:41.328828 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.328684    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-token\" (UniqueName: \"kubernetes.io/projected/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-token\") pod \"182e059b-08d3-4ea4-9a37-3bdc44581625\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") "
Nov 01 06:53:41.328828 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.328696    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-credential-socket" pod "182e059b-08d3-4ea4-9a37-3bdc44581625" (UID: "182e059b-08d3-4ea4-9a37-3bdc44581625"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:53:41.328828 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.328719    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-data\") pod \"182e059b-08d3-4ea4-9a37-3bdc44581625\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") "
Nov 01 06:53:41.328828 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.328762    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-workload-certs\") pod \"182e059b-08d3-4ea4-9a37-3bdc44581625\" (UID: \"182e059b-08d3-4ea4-9a37-3bdc44581625\") "
Nov 01 06:53:41.329099 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.328863    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-workload-socket" pod "182e059b-08d3-4ea4-9a37-3bdc44581625" (UID: "182e059b-08d3-4ea4-9a37-3bdc44581625"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:53:41.329162 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.329093    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-workload-certs" pod "182e059b-08d3-4ea4-9a37-3bdc44581625" (UID: "182e059b-08d3-4ea4-9a37-3bdc44581625"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:53:41.329162 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.329134    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-data" pod "182e059b-08d3-4ea4-9a37-3bdc44581625" (UID: "182e059b-08d3-4ea4-9a37-3bdc44581625"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:53:41.329272 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.329154    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/182e059b-08d3-4ea4-9a37-3bdc44581625-istiod-ca-cert" pod "182e059b-08d3-4ea4-9a37-3bdc44581625" (UID: "182e059b-08d3-4ea4-9a37-3bdc44581625"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:53:41.329272 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.329207    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:41.329272 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.329226    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:41.329272 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.329242    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:41.330746 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.330719    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-podinfo" pod "182e059b-08d3-4ea4-9a37-3bdc44581625" (UID: "182e059b-08d3-4ea4-9a37-3bdc44581625"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:53:41.330862 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.330804    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-envoy" pod "182e059b-08d3-4ea4-9a37-3bdc44581625" (UID: "182e059b-08d3-4ea4-9a37-3bdc44581625"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:53:41.330923 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.330875    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182e059b-08d3-4ea4-9a37-3bdc44581625-kube-api-access-z2hvt" pod "182e059b-08d3-4ea4-9a37-3bdc44581625" (UID: "182e059b-08d3-4ea4-9a37-3bdc44581625"). InnerVolumeSpecName "kube-api-access-z2hvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:53:41.331023 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.331003    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-token" pod "182e059b-08d3-4ea4-9a37-3bdc44581625" (UID: "182e059b-08d3-4ea4-9a37-3bdc44581625"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:53:41.429928 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.429900    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2hvt\" (UniqueName: \"kubernetes.io/projected/182e059b-08d3-4ea4-9a37-3bdc44581625-kube-api-access-z2hvt\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:41.429928 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.429925    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:41.429928 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.429936    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/182e059b-08d3-4ea4-9a37-3bdc44581625-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:41.430225 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.429945    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:41.430225 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.429954    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:41.430225 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.429963    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/182e059b-08d3-4ea4-9a37-3bdc44581625-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:41.507171 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.507084    5619 generic.go:356] "Generic (PLEG): container finished" podID="182e059b-08d3-4ea4-9a37-3bdc44581625" containerID="54e1ee34a189fd8676ef0e56cc458ef698f2f68e688776e17ccdfdb07fe39075" exitCode=0
Nov 01 06:53:41.507171 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.507129    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb" event={"ID":"182e059b-08d3-4ea4-9a37-3bdc44581625","Type":"ContainerDied","Data":"54e1ee34a189fd8676ef0e56cc458ef698f2f68e688776e17ccdfdb07fe39075"}
Nov 01 06:53:41.507377 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.507177    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb" event={"ID":"182e059b-08d3-4ea4-9a37-3bdc44581625","Type":"ContainerDied","Data":"00cb8d76224c4ab5234db223478196294a3acd500eb806a69b6d0f5b44cdc6f7"}
Nov 01 06:53:41.507377 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.507206    5619 scope.go:122] "RemoveContainer" containerID="54e1ee34a189fd8676ef0e56cc458ef698f2f68e688776e17ccdfdb07fe39075"
Nov 01 06:53:41.507377 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.507132    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"
Nov 01 06:53:41.517499 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.517476    5619 scope.go:122] "RemoveContainer" containerID="60da1d2de41805be3601e18106624ba7eea709026d5469985898569c8091de49"
Nov 01 06:53:41.525851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.525833    5619 scope.go:122] "RemoveContainer" containerID="54e1ee34a189fd8676ef0e56cc458ef698f2f68e688776e17ccdfdb07fe39075"
Nov 01 06:53:41.526142 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:53:41.526117    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e1ee34a189fd8676ef0e56cc458ef698f2f68e688776e17ccdfdb07fe39075\": container with ID starting with 54e1ee34a189fd8676ef0e56cc458ef698f2f68e688776e17ccdfdb07fe39075 not found: ID does not exist" containerID="54e1ee34a189fd8676ef0e56cc458ef698f2f68e688776e17ccdfdb07fe39075"
Nov 01 06:53:41.526214 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.526154    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e1ee34a189fd8676ef0e56cc458ef698f2f68e688776e17ccdfdb07fe39075"} err="failed to get container status \"54e1ee34a189fd8676ef0e56cc458ef698f2f68e688776e17ccdfdb07fe39075\": rpc error: code = NotFound desc = could not find container \"54e1ee34a189fd8676ef0e56cc458ef698f2f68e688776e17ccdfdb07fe39075\": container with ID starting with 54e1ee34a189fd8676ef0e56cc458ef698f2f68e688776e17ccdfdb07fe39075 not found: ID does not exist"
Nov 01 06:53:41.526214 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.526173    5619 scope.go:122] "RemoveContainer" containerID="60da1d2de41805be3601e18106624ba7eea709026d5469985898569c8091de49"
Nov 01 06:53:41.526449 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:53:41.526426    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60da1d2de41805be3601e18106624ba7eea709026d5469985898569c8091de49\": container with ID starting with 60da1d2de41805be3601e18106624ba7eea709026d5469985898569c8091de49 not found: ID does not exist" containerID="60da1d2de41805be3601e18106624ba7eea709026d5469985898569c8091de49"
Nov 01 06:53:41.526560 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.526456    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60da1d2de41805be3601e18106624ba7eea709026d5469985898569c8091de49"} err="failed to get container status \"60da1d2de41805be3601e18106624ba7eea709026d5469985898569c8091de49\": rpc error: code = NotFound desc = could not find container \"60da1d2de41805be3601e18106624ba7eea709026d5469985898569c8091de49\": container with ID starting with 60da1d2de41805be3601e18106624ba7eea709026d5469985898569c8091de49 not found: ID does not exist"
Nov 01 06:53:41.550995 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.550969    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:41.675522 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.675489    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"]
Nov 01 06:53:41.681962 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.681936    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-99e79c-openshift-default-789b645f9c-r8nlb"]
Nov 01 06:53:41.857554 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:41.857524    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:42.961697 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:42.961660    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="182e059b-08d3-4ea4-9a37-3bdc44581625" path="/var/lib/kubelet/pods/182e059b-08d3-4ea4-9a37-3bdc44581625/volumes"
Nov 01 06:53:43.394013 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:43.393970    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:44.415764 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:44.415724    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"]
Nov 01 06:53:44.416175 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:44.415933    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg" podUID="4bbaa266-0be4-4403-bbb2-8d4b552899c0" containerName="maas-api" containerID="cri-o://0bfc866e41a016e3574977ac0a1cdf1f7439db5d646e7745080d34ebbb531853" gracePeriod=30
Nov 01 06:53:44.517502 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:44.517467    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:44.670154 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:44.670078    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"
Nov 01 06:53:44.761274 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:44.761236    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bbaa266-0be4-4403-bbb2-8d4b552899c0-kube-api-access-x45pj\" (UniqueName: \"kubernetes.io/projected/4bbaa266-0be4-4403-bbb2-8d4b552899c0-kube-api-access-x45pj\") pod \"4bbaa266-0be4-4403-bbb2-8d4b552899c0\" (UID: \"4bbaa266-0be4-4403-bbb2-8d4b552899c0\") "
Nov 01 06:53:44.761491 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:44.761294    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bbaa266-0be4-4403-bbb2-8d4b552899c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbaa266-0be4-4403-bbb2-8d4b552899c0-metrics-certs\") pod \"4bbaa266-0be4-4403-bbb2-8d4b552899c0\" (UID: \"4bbaa266-0be4-4403-bbb2-8d4b552899c0\") "
Nov 01 06:53:44.761491 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:44.761364    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bbaa266-0be4-4403-bbb2-8d4b552899c0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bbaa266-0be4-4403-bbb2-8d4b552899c0-openshift-service-ca\") pod \"4bbaa266-0be4-4403-bbb2-8d4b552899c0\" (UID: \"4bbaa266-0be4-4403-bbb2-8d4b552899c0\") "
Nov 01 06:53:44.761491 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:44.761452    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bbaa266-0be4-4403-bbb2-8d4b552899c0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bbaa266-0be4-4403-bbb2-8d4b552899c0-maas-api-tls\") pod \"4bbaa266-0be4-4403-bbb2-8d4b552899c0\" (UID: \"4bbaa266-0be4-4403-bbb2-8d4b552899c0\") "
Nov 01 06:53:44.761826 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:44.761745    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bbaa266-0be4-4403-bbb2-8d4b552899c0-openshift-service-ca" pod "4bbaa266-0be4-4403-bbb2-8d4b552899c0" (UID: "4bbaa266-0be4-4403-bbb2-8d4b552899c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:53:44.763444 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:44.763408    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bbaa266-0be4-4403-bbb2-8d4b552899c0-kube-api-access-x45pj" pod "4bbaa266-0be4-4403-bbb2-8d4b552899c0" (UID: "4bbaa266-0be4-4403-bbb2-8d4b552899c0"). InnerVolumeSpecName "kube-api-access-x45pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:53:44.763563 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:44.763472    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bbaa266-0be4-4403-bbb2-8d4b552899c0-maas-api-tls" pod "4bbaa266-0be4-4403-bbb2-8d4b552899c0" (UID: "4bbaa266-0be4-4403-bbb2-8d4b552899c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:53:44.763563 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:44.763551    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bbaa266-0be4-4403-bbb2-8d4b552899c0-metrics-certs" pod "4bbaa266-0be4-4403-bbb2-8d4b552899c0" (UID: "4bbaa266-0be4-4403-bbb2-8d4b552899c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:53:44.862669 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:44.862638    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bbaa266-0be4-4403-bbb2-8d4b552899c0-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:44.862669 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:44.862666    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bbaa266-0be4-4403-bbb2-8d4b552899c0-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:44.862669 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:44.862676    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x45pj\" (UniqueName: \"kubernetes.io/projected/4bbaa266-0be4-4403-bbb2-8d4b552899c0-kube-api-access-x45pj\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:44.862909 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:44.862686    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbaa266-0be4-4403-bbb2-8d4b552899c0-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:53:45.528577 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:45.528528    5619 generic.go:356] "Generic (PLEG): container finished" podID="4bbaa266-0be4-4403-bbb2-8d4b552899c0" containerID="0bfc866e41a016e3574977ac0a1cdf1f7439db5d646e7745080d34ebbb531853" exitCode=0
Nov 01 06:53:45.529064 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:45.528584    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg" event={"ID":"4bbaa266-0be4-4403-bbb2-8d4b552899c0","Type":"ContainerDied","Data":"0bfc866e41a016e3574977ac0a1cdf1f7439db5d646e7745080d34ebbb531853"}
Nov 01 06:53:45.529064 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:45.528635    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg" event={"ID":"4bbaa266-0be4-4403-bbb2-8d4b552899c0","Type":"ContainerDied","Data":"e784625c3e5801b121535f984c2f010902bf0706241c106ad39309751318ecd2"}
Nov 01 06:53:45.529064 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:45.528587    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"
Nov 01 06:53:45.529064 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:45.528666    5619 scope.go:122] "RemoveContainer" containerID="0bfc866e41a016e3574977ac0a1cdf1f7439db5d646e7745080d34ebbb531853"
Nov 01 06:53:45.537888 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:45.537857    5619 scope.go:122] "RemoveContainer" containerID="0bfc866e41a016e3574977ac0a1cdf1f7439db5d646e7745080d34ebbb531853"
Nov 01 06:53:45.538169 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:53:45.538151    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bfc866e41a016e3574977ac0a1cdf1f7439db5d646e7745080d34ebbb531853\": container with ID starting with 0bfc866e41a016e3574977ac0a1cdf1f7439db5d646e7745080d34ebbb531853 not found: ID does not exist" containerID="0bfc866e41a016e3574977ac0a1cdf1f7439db5d646e7745080d34ebbb531853"
Nov 01 06:53:45.538232 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:45.538178    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bfc866e41a016e3574977ac0a1cdf1f7439db5d646e7745080d34ebbb531853"} err="failed to get container status \"0bfc866e41a016e3574977ac0a1cdf1f7439db5d646e7745080d34ebbb531853\": rpc error: code = NotFound desc = could not find container \"0bfc866e41a016e3574977ac0a1cdf1f7439db5d646e7745080d34ebbb531853\": container with ID starting with 0bfc866e41a016e3574977ac0a1cdf1f7439db5d646e7745080d34ebbb531853 not found: ID does not exist"
Nov 01 06:53:45.695949 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:45.695915    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"]
Nov 01 06:53:45.746668 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:45.746634    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3986bf-5f5586dd4f-7rfqg"]
Nov 01 06:53:46.281514 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:46.281478    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:46.961023 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:46.960989    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bbaa266-0be4-4403-bbb2-8d4b552899c0" path="/var/lib/kubelet/pods/4bbaa266-0be4-4403-bbb2-8d4b552899c0/volumes"
Nov 01 06:53:47.841289 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:47.841254    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:47.992634 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:47.992597    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:49.491078 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:49.491051    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:50.627567 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:50.627530    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:50.819253 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:50.819208    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:55.265425 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:55.265350    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:55.878357 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:53:55.878304    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:53:59.378362 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:59.378312    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:53:59.872643 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:53:59.872604    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:00.434216 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:00.434179    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"]
Nov 01 06:54:00.434658 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:00.434422    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s" podUID="b960f047-0688-4d8d-998b-9bc0f5b595b1" containerName="maas-api" containerID="cri-o://8b2fb0045a11ae53db06ee12ed39d74f80ba71ce38f96a589f374321087d66e1" gracePeriod=30
Nov 01 06:54:00.535202 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:00.535165    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:00.605123 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:00.605090    5619 generic.go:356] "Generic (PLEG): container finished" podID="b960f047-0688-4d8d-998b-9bc0f5b595b1" containerID="8b2fb0045a11ae53db06ee12ed39d74f80ba71ce38f96a589f374321087d66e1" exitCode=0
Nov 01 06:54:00.605274 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:00.605256    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s" event={"ID":"b960f047-0688-4d8d-998b-9bc0f5b595b1","Type":"ContainerDied","Data":"8b2fb0045a11ae53db06ee12ed39d74f80ba71ce38f96a589f374321087d66e1"}
Nov 01 06:54:00.699254 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:00.699185    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"
Nov 01 06:54:00.830369 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:00.830333    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b960f047-0688-4d8d-998b-9bc0f5b595b1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b960f047-0688-4d8d-998b-9bc0f5b595b1-metrics-certs\") pod \"b960f047-0688-4d8d-998b-9bc0f5b595b1\" (UID: \"b960f047-0688-4d8d-998b-9bc0f5b595b1\") "
Nov 01 06:54:00.830556 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:00.830408    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b960f047-0688-4d8d-998b-9bc0f5b595b1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b960f047-0688-4d8d-998b-9bc0f5b595b1-openshift-service-ca\") pod \"b960f047-0688-4d8d-998b-9bc0f5b595b1\" (UID: \"b960f047-0688-4d8d-998b-9bc0f5b595b1\") "
Nov 01 06:54:00.830556 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:00.830438    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b960f047-0688-4d8d-998b-9bc0f5b595b1-kube-api-access-p5zk2\" (UniqueName: \"kubernetes.io/projected/b960f047-0688-4d8d-998b-9bc0f5b595b1-kube-api-access-p5zk2\") pod \"b960f047-0688-4d8d-998b-9bc0f5b595b1\" (UID: \"b960f047-0688-4d8d-998b-9bc0f5b595b1\") "
Nov 01 06:54:00.830556 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:00.830462    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b960f047-0688-4d8d-998b-9bc0f5b595b1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b960f047-0688-4d8d-998b-9bc0f5b595b1-maas-api-tls\") pod \"b960f047-0688-4d8d-998b-9bc0f5b595b1\" (UID: \"b960f047-0688-4d8d-998b-9bc0f5b595b1\") "
Nov 01 06:54:00.830873 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:00.830834    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b960f047-0688-4d8d-998b-9bc0f5b595b1-openshift-service-ca" pod "b960f047-0688-4d8d-998b-9bc0f5b595b1" (UID: "b960f047-0688-4d8d-998b-9bc0f5b595b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:54:00.831006 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:00.830937    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b960f047-0688-4d8d-998b-9bc0f5b595b1-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:00.832561 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:00.832538    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b960f047-0688-4d8d-998b-9bc0f5b595b1-maas-api-tls" pod "b960f047-0688-4d8d-998b-9bc0f5b595b1" (UID: "b960f047-0688-4d8d-998b-9bc0f5b595b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:54:00.832681 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:00.832629    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b960f047-0688-4d8d-998b-9bc0f5b595b1-metrics-certs" pod "b960f047-0688-4d8d-998b-9bc0f5b595b1" (UID: "b960f047-0688-4d8d-998b-9bc0f5b595b1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:54:00.832962 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:00.832940    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b960f047-0688-4d8d-998b-9bc0f5b595b1-kube-api-access-p5zk2" pod "b960f047-0688-4d8d-998b-9bc0f5b595b1" (UID: "b960f047-0688-4d8d-998b-9bc0f5b595b1"). InnerVolumeSpecName "kube-api-access-p5zk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:54:00.931644 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:00.931607    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b960f047-0688-4d8d-998b-9bc0f5b595b1-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:00.931644 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:00.931640    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5zk2\" (UniqueName: \"kubernetes.io/projected/b960f047-0688-4d8d-998b-9bc0f5b595b1-kube-api-access-p5zk2\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:00.931644 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:00.931652    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b960f047-0688-4d8d-998b-9bc0f5b595b1-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:01.615471 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:01.615441    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"
Nov 01 06:54:01.615471 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:01.615448    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s" event={"ID":"b960f047-0688-4d8d-998b-9bc0f5b595b1","Type":"ContainerDied","Data":"a7c5b69103055fed6bb69910a6e79ee2c47cd0c790ecf6c3a54d9ae06db2020f"}
Nov 01 06:54:01.615993 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:01.615524    5619 scope.go:122] "RemoveContainer" containerID="8b2fb0045a11ae53db06ee12ed39d74f80ba71ce38f96a589f374321087d66e1"
Nov 01 06:54:01.648228 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:01.648192    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:01.781512 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:01.781474    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"]
Nov 01 06:54:01.832298 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:01.832266    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9c8fc5-6c68d69c9-9sf2s"]
Nov 01 06:54:02.960854 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:02.960822    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b960f047-0688-4d8d-998b-9bc0f5b595b1" path="/var/lib/kubelet/pods/b960f047-0688-4d8d-998b-9bc0f5b595b1/volumes"
Nov 01 06:54:07.629839 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:07.629803    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:07.830215 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:07.830178    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:12.763562 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:12.763523    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:18.247519 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:18.247479    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:18.444273 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:18.444237    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:19.342869 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:19.342828    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:24.273045 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273007    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"]
Nov 01 06:54:24.273453 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273427    5619 cpu_manager.go:414] "containerMap: removing container" podUID="22cb8e7f-72a8-4cf7-a419-8470c27d8799" containerName="payload-processing"
Nov 01 06:54:24.273453 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273440    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22cb8e7f-72a8-4cf7-a419-8470c27d8799" containerName="payload-processing"
Nov 01 06:54:24.273538 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273455    5619 cpu_manager.go:414] "containerMap: removing container" podUID="c9089f8a-a6b6-4c53-b483-8b5991330c33" containerName="maas-api"
Nov 01 06:54:24.273538 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273465    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9089f8a-a6b6-4c53-b483-8b5991330c33" containerName="maas-api"
Nov 01 06:54:24.273538 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273471    5619 cpu_manager.go:414] "containerMap: removing container" podUID="182e059b-08d3-4ea4-9a37-3bdc44581625" containerName="istio-proxy"
Nov 01 06:54:24.273538 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273477    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182e059b-08d3-4ea4-9a37-3bdc44581625" containerName="istio-proxy"
Nov 01 06:54:24.273538 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273486    5619 cpu_manager.go:414] "containerMap: removing container" podUID="df8c3448-d616-414e-8e09-d39e82928731" containerName="payload-pre-processing"
Nov 01 06:54:24.273538 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273492    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df8c3448-d616-414e-8e09-d39e82928731" containerName="payload-pre-processing"
Nov 01 06:54:24.273538 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273502    5619 cpu_manager.go:414] "containerMap: removing container" podUID="182e059b-08d3-4ea4-9a37-3bdc44581625" containerName="istio-proxy"
Nov 01 06:54:24.273538 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273508    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182e059b-08d3-4ea4-9a37-3bdc44581625" containerName="istio-proxy"
Nov 01 06:54:24.273538 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273517    5619 cpu_manager.go:414] "containerMap: removing container" podUID="b960f047-0688-4d8d-998b-9bc0f5b595b1" containerName="maas-api"
Nov 01 06:54:24.273538 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273522    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b960f047-0688-4d8d-998b-9bc0f5b595b1" containerName="maas-api"
Nov 01 06:54:24.273538 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273528    5619 cpu_manager.go:414] "containerMap: removing container" podUID="a8b15032-f506-448c-a9a6-f5f45930f3a6" containerName="payload-processing"
Nov 01 06:54:24.273538 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273535    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b15032-f506-448c-a9a6-f5f45930f3a6" containerName="payload-processing"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273548    5619 cpu_manager.go:414] "containerMap: removing container" podUID="1b1d6695-9e95-46fe-b1c4-f6f4b7573f67" containerName="maas-api"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273553    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b1d6695-9e95-46fe-b1c4-f6f4b7573f67" containerName="maas-api"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273560    5619 cpu_manager.go:414] "containerMap: removing container" podUID="4bbaa266-0be4-4403-bbb2-8d4b552899c0" containerName="maas-api"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273564    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bbaa266-0be4-4403-bbb2-8d4b552899c0" containerName="maas-api"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273578    5619 cpu_manager.go:414] "containerMap: removing container" podUID="4482833c-590d-44a1-a485-651970b9abf6" containerName="istio-proxy"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273583    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4482833c-590d-44a1-a485-651970b9abf6" containerName="istio-proxy"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273591    5619 cpu_manager.go:414] "containerMap: removing container" podUID="bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" containerName="istio-proxy"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273595    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" containerName="istio-proxy"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273603    5619 cpu_manager.go:414] "containerMap: removing container" podUID="4cc4e41a-b8cb-4d9c-a1a6-07c701659943" containerName="payload-processing"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273610    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc4e41a-b8cb-4d9c-a1a6-07c701659943" containerName="payload-processing"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273619    5619 cpu_manager.go:414] "containerMap: removing container" podUID="69cd086b-554c-49c9-a471-831d40610ceb" containerName="istio-proxy"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273624    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69cd086b-554c-49c9-a471-831d40610ceb" containerName="istio-proxy"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273630    5619 cpu_manager.go:414] "containerMap: removing container" podUID="6b322967-2ec4-4195-b5dc-9524a46b5091" containerName="payload-processing"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273635    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b322967-2ec4-4195-b5dc-9524a46b5091" containerName="payload-processing"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273643    5619 cpu_manager.go:414] "containerMap: removing container" podUID="7f4f13d7-922b-4478-afe8-0b9c44d894ce" containerName="payload-pre-processing"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273648    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4f13d7-922b-4478-afe8-0b9c44d894ce" containerName="payload-pre-processing"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273657    5619 cpu_manager.go:414] "containerMap: removing container" podUID="1e3999df-3132-435d-93b4-0ed000180c4d" containerName="maas-api"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273661    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e3999df-3132-435d-93b4-0ed000180c4d" containerName="maas-api"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273666    5619 cpu_manager.go:414] "containerMap: removing container" podUID="dbd66390-4712-4e35-acae-26c5b43e35ef" containerName="payload-pre-processing"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273671    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd66390-4712-4e35-acae-26c5b43e35ef" containerName="payload-pre-processing"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273680    5619 cpu_manager.go:414] "containerMap: removing container" podUID="96534e12-7487-48ec-bfaa-9d7cc14181d2" containerName="istio-proxy"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273687    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96534e12-7487-48ec-bfaa-9d7cc14181d2" containerName="istio-proxy"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273697    5619 cpu_manager.go:414] "containerMap: removing container" podUID="e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6" containerName="payload-pre-processing"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273702    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6" containerName="payload-pre-processing"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273713    5619 cpu_manager.go:414] "containerMap: removing container" podUID="21f94020-a328-405d-8604-81102600dbb2" containerName="maas-api"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273717    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21f94020-a328-405d-8604-81102600dbb2" containerName="maas-api"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273724    5619 cpu_manager.go:414] "containerMap: removing container" podUID="d3e57167-aba3-406a-9a21-a48ae4584959" containerName="payload-pre-processing"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273728    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e57167-aba3-406a-9a21-a48ae4584959" containerName="payload-pre-processing"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273736    5619 cpu_manager.go:414] "containerMap: removing container" podUID="7ec112e9-286e-42f5-be40-f6954033585c" containerName="payload-processing"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273740    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec112e9-286e-42f5-be40-f6954033585c" containerName="payload-processing"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273748    5619 cpu_manager.go:414] "containerMap: removing container" podUID="5fa0499e-e650-4321-9162-ca5b191f1ec5" containerName="maas-api"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273752    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa0499e-e650-4321-9162-ca5b191f1ec5" containerName="maas-api"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273804    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="96534e12-7487-48ec-bfaa-9d7cc14181d2" containerName="istio-proxy"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273814    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="e49d0f57-4d8e-45fc-9f7b-2d37e5d731c6" containerName="payload-pre-processing"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273821    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e3999df-3132-435d-93b4-0ed000180c4d" containerName="maas-api"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273829    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9089f8a-a6b6-4c53-b483-8b5991330c33" containerName="maas-api"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273836    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="21f94020-a328-405d-8604-81102600dbb2" containerName="maas-api"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273845    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e57167-aba3-406a-9a21-a48ae4584959" containerName="payload-pre-processing"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273853    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec112e9-286e-42f5-be40-f6954033585c" containerName="payload-processing"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273857    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa0499e-e650-4321-9162-ca5b191f1ec5" containerName="maas-api"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273863    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="22cb8e7f-72a8-4cf7-a419-8470c27d8799" containerName="payload-processing"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273869    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="182e059b-08d3-4ea4-9a37-3bdc44581625" containerName="istio-proxy"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273873    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9089f8a-a6b6-4c53-b483-8b5991330c33" containerName="maas-api"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273878    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="df8c3448-d616-414e-8e09-d39e82928731" containerName="payload-pre-processing"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273883    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="182e059b-08d3-4ea4-9a37-3bdc44581625" containerName="istio-proxy"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273888    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="b960f047-0688-4d8d-998b-9bc0f5b595b1" containerName="maas-api"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273893    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b15032-f506-448c-a9a6-f5f45930f3a6" containerName="payload-processing"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273898    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b1d6695-9e95-46fe-b1c4-f6f4b7573f67" containerName="maas-api"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273903    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bbaa266-0be4-4403-bbb2-8d4b552899c0" containerName="maas-api"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273912    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="4482833c-590d-44a1-a485-651970b9abf6" containerName="istio-proxy"
Nov 01 06:54:24.273899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273921    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7114a9-2bb2-44ec-aa8d-11a9b7cc0cea" containerName="istio-proxy"
Nov 01 06:54:24.275690 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273929    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="69cd086b-554c-49c9-a471-831d40610ceb" containerName="istio-proxy"
Nov 01 06:54:24.275690 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273935    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc4e41a-b8cb-4d9c-a1a6-07c701659943" containerName="payload-processing"
Nov 01 06:54:24.275690 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273939    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b322967-2ec4-4195-b5dc-9524a46b5091" containerName="payload-processing"
Nov 01 06:54:24.275690 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273945    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f4f13d7-922b-4478-afe8-0b9c44d894ce" containerName="payload-pre-processing"
Nov 01 06:54:24.275690 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.273950    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd66390-4712-4e35-acae-26c5b43e35ef" containerName="payload-pre-processing"
Nov 01 06:54:24.276893 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.276877    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.330162 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.330122    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9c8fc5-openshift-default-dockercfg-6mn84\""
Nov 01 06:54:24.336551 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.336523    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01339151-0844-4b39-aebc-93cabc4afd86-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.336691 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.336590    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.336691 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.336662    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01339151-0844-4b39-aebc-93cabc4afd86-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.336797 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.336698    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.336797 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.336730    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.336881 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.336804    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.336881 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.336850    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01339151-0844-4b39-aebc-93cabc4afd86-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.336881 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.336871    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh4wc\" (UniqueName: \"kubernetes.io/projected/01339151-0844-4b39-aebc-93cabc4afd86-kube-api-access-lh4wc\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.336999 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.336904    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.382507 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.382477    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"]
Nov 01 06:54:24.437357 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.437316    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.437542 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.437369    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.437542 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.437390    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.437542 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.437412    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01339151-0844-4b39-aebc-93cabc4afd86-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.437542 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.437427    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh4wc\" (UniqueName: \"kubernetes.io/projected/01339151-0844-4b39-aebc-93cabc4afd86-kube-api-access-lh4wc\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.437542 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.437458    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.437542 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.437480    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01339151-0844-4b39-aebc-93cabc4afd86-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.437876 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.437547    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.437876 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.437601    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01339151-0844-4b39-aebc-93cabc4afd86-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.437876 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.437775    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-istio-data\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.438137 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.438102    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01339151-0844-4b39-aebc-93cabc4afd86-istiod-ca-cert\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.438234 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.438158    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-workload-certs\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.438234 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.438190    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-workload-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.438234 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.438162    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-credential-socket\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.439711 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.439692    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-istio-envoy\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.440000 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.439983    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01339151-0844-4b39-aebc-93cabc4afd86-istio-podinfo\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.499566 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.499538    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01339151-0844-4b39-aebc-93cabc4afd86-istio-token\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.499704 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.499608    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh4wc\" (UniqueName: \"kubernetes.io/projected/01339151-0844-4b39-aebc-93cabc4afd86-kube-api-access-lh4wc\") pod \"e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") " pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:24.589376 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:24.589291    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:25.010555 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:25.010530    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"]
Nov 01 06:54:25.012990 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:54:25.012960    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01339151_0844_4b39_aebc_93cabc4afd86.slice/crio-28bd4fe4fd2b58a7af0bda279b9e706b34429444d9e84197151972e307cf75c6 WatchSource:0}: Error finding container 28bd4fe4fd2b58a7af0bda279b9e706b34429444d9e84197151972e307cf75c6: Status 404 returned error can't find the container with id 28bd4fe4fd2b58a7af0bda279b9e706b34429444d9e84197151972e307cf75c6
Nov 01 06:54:25.015069 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:25.015038    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:54:25.015159 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:25.015122    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:54:25.015159 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:25.015149    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:54:25.741115 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:25.741066    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt" event={"ID":"01339151-0844-4b39-aebc-93cabc4afd86","Type":"ContainerStarted","Data":"1e8def82af40f279e2277159cfed4d33f79c500d51388ec0e1dacc068cc0cd2a"}
Nov 01 06:54:25.741580 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:25.741178    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt" event={"ID":"01339151-0844-4b39-aebc-93cabc4afd86","Type":"ContainerStarted","Data":"28bd4fe4fd2b58a7af0bda279b9e706b34429444d9e84197151972e307cf75c6"}
Nov 01 06:54:25.869122 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:25.869072    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt" podStartSLOduration=1.8690578709999999 podStartE2EDuration="1.869057871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:54:25.868252763 +0000 UTC m=+4103.079076272" watchObservedRunningTime="2026-11-01 06:54:25.869057871 +0000 UTC m=+4103.079881380"
Nov 01 06:54:25.879776 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:54:25.879746    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:54:25.885485 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:25.885461    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:26.082921 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:26.082823    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:26.590047 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:26.590008    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:26.595351 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:26.595311    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:26.745571 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:26.745536    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:26.746598 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:26.746580    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:28.450809 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:28.450774    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:28.597880 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:28.597844    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:28.743192 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:28.743101    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:30.076650 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:30.076614    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:30.569475 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:30.569444    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"]
Nov 01 06:54:30.569695 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:30.569672    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt" podUID="01339151-0844-4b39-aebc-93cabc4afd86" containerName="istio-proxy" containerID="cri-o://1e8def82af40f279e2277159cfed4d33f79c500d51388ec0e1dacc068cc0cd2a" gracePeriod=30
Nov 01 06:54:30.577367 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:30.577338    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:34.127097 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:34.127062    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-zxf6q_b3857be3-6a36-43f7-adec-7c924b444dd1/manager/0.log"
Nov 01 06:54:35.450522 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.450488    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:35.796686 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.796652    5619 generic.go:356] "Generic (PLEG): container finished" podID="01339151-0844-4b39-aebc-93cabc4afd86" containerID="1e8def82af40f279e2277159cfed4d33f79c500d51388ec0e1dacc068cc0cd2a" exitCode=0
Nov 01 06:54:35.796851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.796724    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt" event={"ID":"01339151-0844-4b39-aebc-93cabc4afd86","Type":"ContainerDied","Data":"1e8def82af40f279e2277159cfed4d33f79c500d51388ec0e1dacc068cc0cd2a"}
Nov 01 06:54:35.821942 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.821917    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:35.944105 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.944071    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01339151-0844-4b39-aebc-93cabc4afd86-kube-api-access-lh4wc\" (UniqueName: \"kubernetes.io/projected/01339151-0844-4b39-aebc-93cabc4afd86-kube-api-access-lh4wc\") pod \"01339151-0844-4b39-aebc-93cabc4afd86\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") "
Nov 01 06:54:35.944299 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.944134    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-istio-data\") pod \"01339151-0844-4b39-aebc-93cabc4afd86\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") "
Nov 01 06:54:35.944299 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.944192    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-credential-socket\") pod \"01339151-0844-4b39-aebc-93cabc4afd86\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") "
Nov 01 06:54:35.944299 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.944233    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01339151-0844-4b39-aebc-93cabc4afd86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01339151-0844-4b39-aebc-93cabc4afd86-istiod-ca-cert\") pod \"01339151-0844-4b39-aebc-93cabc4afd86\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") "
Nov 01 06:54:35.944299 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.944261    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-istio-envoy\") pod \"01339151-0844-4b39-aebc-93cabc4afd86\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") "
Nov 01 06:54:35.944542 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.944312    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01339151-0844-4b39-aebc-93cabc4afd86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01339151-0844-4b39-aebc-93cabc4afd86-istio-podinfo\") pod \"01339151-0844-4b39-aebc-93cabc4afd86\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") "
Nov 01 06:54:35.944542 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.944370    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-workload-socket\") pod \"01339151-0844-4b39-aebc-93cabc4afd86\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") "
Nov 01 06:54:35.944849 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.944608    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-istio-data" pod "01339151-0844-4b39-aebc-93cabc4afd86" (UID: "01339151-0844-4b39-aebc-93cabc4afd86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:54:35.944949 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.944920    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-credential-socket" pod "01339151-0844-4b39-aebc-93cabc4afd86" (UID: "01339151-0844-4b39-aebc-93cabc4afd86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:54:35.945771 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.945230    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01339151-0844-4b39-aebc-93cabc4afd86-istiod-ca-cert" pod "01339151-0844-4b39-aebc-93cabc4afd86" (UID: "01339151-0844-4b39-aebc-93cabc4afd86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 06:54:35.945771 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.945237    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-workload-socket" pod "01339151-0844-4b39-aebc-93cabc4afd86" (UID: "01339151-0844-4b39-aebc-93cabc4afd86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:54:35.945771 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.944439    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01339151-0844-4b39-aebc-93cabc4afd86-istio-token\" (UniqueName: \"kubernetes.io/projected/01339151-0844-4b39-aebc-93cabc4afd86-istio-token\") pod \"01339151-0844-4b39-aebc-93cabc4afd86\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") "
Nov 01 06:54:35.945771 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.945352    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-workload-certs\") pod \"01339151-0844-4b39-aebc-93cabc4afd86\" (UID: \"01339151-0844-4b39-aebc-93cabc4afd86\") "
Nov 01 06:54:35.945771 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.945734    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-workload-certs" pod "01339151-0844-4b39-aebc-93cabc4afd86" (UID: "01339151-0844-4b39-aebc-93cabc4afd86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:54:35.946108 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.946030    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01339151-0844-4b39-aebc-93cabc4afd86-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:35.946108 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.946052    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:35.946108 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.946068    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:35.946108 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.946083    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:35.946108 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.946104    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:35.948868 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.947686    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-istio-envoy" pod "01339151-0844-4b39-aebc-93cabc4afd86" (UID: "01339151-0844-4b39-aebc-93cabc4afd86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:54:35.949404 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.949366    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01339151-0844-4b39-aebc-93cabc4afd86-istio-podinfo" pod "01339151-0844-4b39-aebc-93cabc4afd86" (UID: "01339151-0844-4b39-aebc-93cabc4afd86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 06:54:35.955364 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.955335    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01339151-0844-4b39-aebc-93cabc4afd86-kube-api-access-lh4wc" pod "01339151-0844-4b39-aebc-93cabc4afd86" (UID: "01339151-0844-4b39-aebc-93cabc4afd86"). InnerVolumeSpecName "kube-api-access-lh4wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:54:35.955364 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:35.955352    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01339151-0844-4b39-aebc-93cabc4afd86-istio-token" pod "01339151-0844-4b39-aebc-93cabc4afd86" (UID: "01339151-0844-4b39-aebc-93cabc4afd86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:54:36.046667 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:36.046590    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01339151-0844-4b39-aebc-93cabc4afd86-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:36.046667 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:36.046622    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01339151-0844-4b39-aebc-93cabc4afd86-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:36.046667 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:36.046635    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh4wc\" (UniqueName: \"kubernetes.io/projected/01339151-0844-4b39-aebc-93cabc4afd86-kube-api-access-lh4wc\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:36.046667 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:36.046650    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01339151-0844-4b39-aebc-93cabc4afd86-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:54:36.802550 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:36.802511    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt" event={"ID":"01339151-0844-4b39-aebc-93cabc4afd86","Type":"ContainerDied","Data":"28bd4fe4fd2b58a7af0bda279b9e706b34429444d9e84197151972e307cf75c6"}
Nov 01 06:54:36.803013 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:36.802576    5619 scope.go:122] "RemoveContainer" containerID="1e8def82af40f279e2277159cfed4d33f79c500d51388ec0e1dacc068cc0cd2a"
Nov 01 06:54:36.803013 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:36.802523    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"
Nov 01 06:54:36.986733 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:36.986699    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"]
Nov 01 06:54:37.037415 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:37.037386    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9c8fc5-openshift-default-848d8974f7-gqjnt"]
Nov 01 06:54:38.960901 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:38.960862    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01339151-0844-4b39-aebc-93cabc4afd86" path="/var/lib/kubelet/pods/01339151-0844-4b39-aebc-93cabc4afd86/volumes"
Nov 01 06:54:39.855803 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:39.855768    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:44.928502 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:44.928460    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:45.768145 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:45.768108    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:46.268680 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:46.268644    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-zgwzb_02fae5ae-abd3-4895-a525-42ff3bae5ff4/manager/0.log"
Nov 01 06:54:47.684831 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:47.684797    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bf8b5db8-v4x7w_f577a21b-1c78-4528-a1ea-a6a34c77260c/manager/1.log"
Nov 01 06:54:47.918468 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:47.918442    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-zvcc2_9d763214-1e93-47b0-bfb4-e3321fc6d742/server/0.log"
Nov 01 06:54:49.405565 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:49.405534    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bf8b5db8-v4x7w_f577a21b-1c78-4528-a1ea-a6a34c77260c/manager/0.log"
Nov 01 06:54:53.842035 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:53.842004    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:54.951303 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:54.951267    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"]
Nov 01 06:54:54.951700 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:54.951674    5619 cpu_manager.go:414] "containerMap: removing container" podUID="c9089f8a-a6b6-4c53-b483-8b5991330c33" containerName="maas-api"
Nov 01 06:54:54.951700 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:54.951688    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9089f8a-a6b6-4c53-b483-8b5991330c33" containerName="maas-api"
Nov 01 06:54:54.951780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:54.951714    5619 cpu_manager.go:414] "containerMap: removing container" podUID="01339151-0844-4b39-aebc-93cabc4afd86" containerName="istio-proxy"
Nov 01 06:54:54.951780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:54.951721    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01339151-0844-4b39-aebc-93cabc4afd86" containerName="istio-proxy"
Nov 01 06:54:54.951780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:54.951729    5619 cpu_manager.go:414] "containerMap: removing container" podUID="4482833c-590d-44a1-a485-651970b9abf6" containerName="istio-proxy"
Nov 01 06:54:54.951780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:54.951735    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4482833c-590d-44a1-a485-651970b9abf6" containerName="istio-proxy"
Nov 01 06:54:54.951780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:54.951751    5619 cpu_manager.go:414] "containerMap: removing container" podUID="1e3999df-3132-435d-93b4-0ed000180c4d" containerName="maas-api"
Nov 01 06:54:54.951780 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:54.951755    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e3999df-3132-435d-93b4-0ed000180c4d" containerName="maas-api"
Nov 01 06:54:54.951972 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:54.951828    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e3999df-3132-435d-93b4-0ed000180c4d" containerName="maas-api"
Nov 01 06:54:54.951972 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:54.951843    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="01339151-0844-4b39-aebc-93cabc4afd86" containerName="istio-proxy"
Nov 01 06:54:54.951972 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:54.951851    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="4482833c-590d-44a1-a485-651970b9abf6" containerName="istio-proxy"
Nov 01 06:54:54.955116 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:54.955089    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.008423 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.008374    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-main-045f69-openshift-default-dockercfg-4slwf\""
Nov 01 06:54:55.033098 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.033064    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-credential-socket\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.033239 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.033118    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-envoy\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.033239 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.033216    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-workload-certs\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.033316 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.033261    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-token\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.033372 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.033345    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-data\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.033410 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.033379    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e79bdaee-9cd1-4204-8f2f-d8d356851307-istiod-ca-cert\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.033457 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.033441    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh72n\" (UniqueName: \"kubernetes.io/projected/e79bdaee-9cd1-4204-8f2f-d8d356851307-kube-api-access-bh72n\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.033494 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.033474    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-podinfo\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.033530 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.033494    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-workload-socket\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.054472 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.054440    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:55.061853 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.061827    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"]
Nov 01 06:54:55.134239 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.134203    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-data\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.134239 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.134244    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e79bdaee-9cd1-4204-8f2f-d8d356851307-istiod-ca-cert\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.134530 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.134281    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh72n\" (UniqueName: \"kubernetes.io/projected/e79bdaee-9cd1-4204-8f2f-d8d356851307-kube-api-access-bh72n\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.134530 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.134299    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-podinfo\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.134530 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.134317    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-workload-socket\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.134530 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.134369    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-credential-socket\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.134530 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.134418    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-envoy\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.134530 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.134510    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-workload-certs\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.134879 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.134580    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-token\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.134879 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.134700    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-data\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.134879 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.134723    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-workload-socket\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.134879 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.134798    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-credential-socket\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.135113 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.134974    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-workload-certs\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.135257 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.135233    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e79bdaee-9cd1-4204-8f2f-d8d356851307-istiod-ca-cert\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.136719 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.136700    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-envoy\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.136776 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.136740    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-podinfo\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.196392 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.196360    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-token\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.196546 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.196477    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh72n\" (UniqueName: \"kubernetes.io/projected/e79bdaee-9cd1-4204-8f2f-d8d356851307-kube-api-access-bh72n\") pod \"e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") " pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.269079 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.269007    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:55.381942 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.381906    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:55.692071 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.692045    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"]
Nov 01 06:54:55.693500 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:54:55.693460    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79bdaee_9cd1_4204_8f2f_d8d356851307.slice/crio-92bc40bb5e75a7bb03314292d2f1cf4267444d6efa97d64def1d31bf30c946d0 WatchSource:0}: Error finding container 92bc40bb5e75a7bb03314292d2f1cf4267444d6efa97d64def1d31bf30c946d0: Status 404 returned error can't find the container with id 92bc40bb5e75a7bb03314292d2f1cf4267444d6efa97d64def1d31bf30c946d0
Nov 01 06:54:55.695664 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.695631    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:54:55.695771 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.695697    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:54:55.695771 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.695725    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:54:55.879042 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:54:55.879008    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:54:55.889633 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.889603    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b" event={"ID":"e79bdaee-9cd1-4204-8f2f-d8d356851307","Type":"ContainerStarted","Data":"6294debcf630b144d3526c3b4ff89e02ed7c2d16ed2c80c34ffd401b881ec457"}
Nov 01 06:54:55.889789 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:55.889652    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b" event={"ID":"e79bdaee-9cd1-4204-8f2f-d8d356851307","Type":"ContainerStarted","Data":"92bc40bb5e75a7bb03314292d2f1cf4267444d6efa97d64def1d31bf30c946d0"}
Nov 01 06:54:56.013862 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:56.013757    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b" podStartSLOduration=2.013736769 podStartE2EDuration="2.013736769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:54:56.011491345 +0000 UTC m=+4133.222314853" watchObservedRunningTime="2026-11-01 06:54:56.013736769 +0000 UTC m=+4133.224560279"
Nov 01 06:54:56.269262 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:56.269184    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:56.274151 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:56.274118    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:56.894276 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:56.894244    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:56.895237 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:56.895215    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:54:58.850446 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:58.850407    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:59.018405 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:59.018369    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:54:59.171860 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:54:59.171773    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:02.122187 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:02.122157    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:02.294229 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:02.294196    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:02.683761 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:02.683726    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7"]
Nov 01 06:55:02.687500 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:02.687481    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7"
Nov 01 06:55:02.707899 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:02.707871    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2ccc\" (UniqueName: \"kubernetes.io/projected/7a56ff1b-d00d-4595-a328-a859dc31b021-kube-api-access-z2ccc\") pod \"payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7\" (UID: \"7a56ff1b-d00d-4595-a328-a859dc31b021\") " pod="openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7"
Nov 01 06:55:02.708024 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:02.707914    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a56ff1b-d00d-4595-a328-a859dc31b021-plugins-config-volume\") pod \"payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7\" (UID: \"7a56ff1b-d00d-4595-a328-a859dc31b021\") " pod="openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7"
Nov 01 06:55:02.743561 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:02.743535    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-main-045f69\""
Nov 01 06:55:02.746217 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:02.746184    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7"]
Nov 01 06:55:02.794565 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:02.794532    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-main-045f69-dockercfg-5bgfw\""
Nov 01 06:55:02.809381 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:02.809346    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2ccc\" (UniqueName: \"kubernetes.io/projected/7a56ff1b-d00d-4595-a328-a859dc31b021-kube-api-access-z2ccc\") pod \"payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7\" (UID: \"7a56ff1b-d00d-4595-a328-a859dc31b021\") " pod="openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7"
Nov 01 06:55:02.809530 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:02.809402    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a56ff1b-d00d-4595-a328-a859dc31b021-plugins-config-volume\") pod \"payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7\" (UID: \"7a56ff1b-d00d-4595-a328-a859dc31b021\") " pod="openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7"
Nov 01 06:55:02.810116 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:02.810093    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a56ff1b-d00d-4595-a328-a859dc31b021-plugins-config-volume\") pod \"payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7\" (UID: \"7a56ff1b-d00d-4595-a328-a859dc31b021\") " pod="openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7"
Nov 01 06:55:02.871344 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:02.871300    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2ccc\" (UniqueName: \"kubernetes.io/projected/7a56ff1b-d00d-4595-a328-a859dc31b021-kube-api-access-z2ccc\") pod \"payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7\" (UID: \"7a56ff1b-d00d-4595-a328-a859dc31b021\") " pod="openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7"
Nov 01 06:55:03.053241 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:03.053165    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-main-045f69-dockercfg-5bgfw\""
Nov 01 06:55:03.061635 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:03.061614    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7"
Nov 01 06:55:03.432710 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:03.432683    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7"]
Nov 01 06:55:03.434643 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:55:03.434610    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a56ff1b_d00d_4595_a328_a859dc31b021.slice/crio-c6efa0a77666837821831cc5e809abfad85c293d71a9b718f1ccbcae45cd094a WatchSource:0}: Error finding container c6efa0a77666837821831cc5e809abfad85c293d71a9b718f1ccbcae45cd094a: Status 404 returned error can't find the container with id c6efa0a77666837821831cc5e809abfad85c293d71a9b718f1ccbcae45cd094a
Nov 01 06:55:03.928773 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:03.928732    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7" event={"ID":"7a56ff1b-d00d-4595-a328-a859dc31b021","Type":"ContainerStarted","Data":"0103b28ff94b1170327f28848747d094c3d9d2675920823e8e112dc185fcaf8d"}
Nov 01 06:55:03.928949 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:03.928810    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7" event={"ID":"7a56ff1b-d00d-4595-a328-a859dc31b021","Type":"ContainerStarted","Data":"c6efa0a77666837821831cc5e809abfad85c293d71a9b718f1ccbcae45cd094a"}
Nov 01 06:55:03.928949 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:03.928853    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7"
Nov 01 06:55:04.048692 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:04.048639    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7" podStartSLOduration=2.048626366 podStartE2EDuration="2.048626366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:55:04.047125397 +0000 UTC m=+4141.257948907" watchObservedRunningTime="2026-11-01 06:55:04.048626366 +0000 UTC m=+4141.259449928"
Nov 01 06:55:04.835081 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:04.835044    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"]
Nov 01 06:55:04.839804 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:04.839785    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"
Nov 01 06:55:04.893063 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:04.893032    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-main-045f69-serving-cert\""
Nov 01 06:55:04.929842 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:04.929809    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwtx7\" (UniqueName: \"kubernetes.io/projected/6fc66d3c-9dc5-4a05-908e-30240252019a-kube-api-access-xwtx7\") pod \"maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq\" (UID: \"6fc66d3c-9dc5-4a05-908e-30240252019a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"
Nov 01 06:55:04.930315 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:04.929858    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fc66d3c-9dc5-4a05-908e-30240252019a-maas-api-tls\") pod \"maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq\" (UID: \"6fc66d3c-9dc5-4a05-908e-30240252019a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"
Nov 01 06:55:04.930315 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:04.929971    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fc66d3c-9dc5-4a05-908e-30240252019a-openshift-service-ca\") pod \"maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq\" (UID: \"6fc66d3c-9dc5-4a05-908e-30240252019a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"
Nov 01 06:55:04.930315 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:04.930085    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fc66d3c-9dc5-4a05-908e-30240252019a-metrics-certs\") pod \"maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq\" (UID: \"6fc66d3c-9dc5-4a05-908e-30240252019a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"
Nov 01 06:55:04.945707 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:04.945681    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"]
Nov 01 06:55:05.030242 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:05.030202    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:05.030661 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:05.030640    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fc66d3c-9dc5-4a05-908e-30240252019a-maas-api-tls\") pod \"maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq\" (UID: \"6fc66d3c-9dc5-4a05-908e-30240252019a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"
Nov 01 06:55:05.030763 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:05.030697    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fc66d3c-9dc5-4a05-908e-30240252019a-openshift-service-ca\") pod \"maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq\" (UID: \"6fc66d3c-9dc5-4a05-908e-30240252019a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"
Nov 01 06:55:05.030828 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:05.030758    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fc66d3c-9dc5-4a05-908e-30240252019a-metrics-certs\") pod \"maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq\" (UID: \"6fc66d3c-9dc5-4a05-908e-30240252019a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"
Nov 01 06:55:05.030828 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:55:05.030772    5619 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-serving-cert: secret "maas-api-e2e-worker-main-045f69-serving-cert" not found
Nov 01 06:55:05.030828 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:55:05.030823    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fc66d3c-9dc5-4a05-908e-30240252019a-maas-api-tls podName:6fc66d3c-9dc5-4a05-908e-30240252019a nodeName:}" failed. No retries permitted until 2026-11-01 06:55:05.530808769 +0000 UTC m=+4142.741632257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6fc66d3c-9dc5-4a05-908e-30240252019a-maas-api-tls") pod "maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq" (UID: "6fc66d3c-9dc5-4a05-908e-30240252019a") : secret "maas-api-e2e-worker-main-045f69-serving-cert" not found
Nov 01 06:55:05.030996 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:05.030903    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwtx7\" (UniqueName: \"kubernetes.io/projected/6fc66d3c-9dc5-4a05-908e-30240252019a-kube-api-access-xwtx7\") pod \"maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq\" (UID: \"6fc66d3c-9dc5-4a05-908e-30240252019a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"
Nov 01 06:55:05.031404 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:05.031379    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fc66d3c-9dc5-4a05-908e-30240252019a-openshift-service-ca\") pod \"maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq\" (UID: \"6fc66d3c-9dc5-4a05-908e-30240252019a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"
Nov 01 06:55:05.033196 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:05.033174    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fc66d3c-9dc5-4a05-908e-30240252019a-metrics-certs\") pod \"maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq\" (UID: \"6fc66d3c-9dc5-4a05-908e-30240252019a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"
Nov 01 06:55:05.095310 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:05.095273    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwtx7\" (UniqueName: \"kubernetes.io/projected/6fc66d3c-9dc5-4a05-908e-30240252019a-kube-api-access-xwtx7\") pod \"maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq\" (UID: \"6fc66d3c-9dc5-4a05-908e-30240252019a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"
Nov 01 06:55:05.186596 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:05.186557    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:05.332102 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:05.332063    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:05.535994 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:05.535910    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fc66d3c-9dc5-4a05-908e-30240252019a-maas-api-tls\") pod \"maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq\" (UID: \"6fc66d3c-9dc5-4a05-908e-30240252019a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"
Nov 01 06:55:05.538267 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:05.538246    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fc66d3c-9dc5-4a05-908e-30240252019a-maas-api-tls\") pod \"maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq\" (UID: \"6fc66d3c-9dc5-4a05-908e-30240252019a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"
Nov 01 06:55:05.752052 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:05.752012    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"
Nov 01 06:55:06.129827 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:06.129802    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"]
Nov 01 06:55:06.132184 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:55:06.132157    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc66d3c_9dc5_4a05_908e_30240252019a.slice/crio-abcd45edb423e1983a246317701cba7e1721e80df89ebf5847e4267e1026e6f9 WatchSource:0}: Error finding container abcd45edb423e1983a246317701cba7e1721e80df89ebf5847e4267e1026e6f9: Status 404 returned error can't find the container with id abcd45edb423e1983a246317701cba7e1721e80df89ebf5847e4267e1026e6f9
Nov 01 06:55:06.133577 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:06.133559    5619 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 06:55:06.944898 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:06.944863    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq" event={"ID":"6fc66d3c-9dc5-4a05-908e-30240252019a","Type":"ContainerStarted","Data":"abcd45edb423e1983a246317701cba7e1721e80df89ebf5847e4267e1026e6f9"}
Nov 01 06:55:07.950979 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:07.950941    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq" event={"ID":"6fc66d3c-9dc5-4a05-908e-30240252019a","Type":"ContainerStarted","Data":"9ce930a06fc232cd6d07cc53caef039feae3b7f99bbe56cbae613c52f23cc116"}
Nov 01 06:55:07.951410 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:07.951077    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"
Nov 01 06:55:08.071656 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:08.071597    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq" podStartSLOduration=3.150851132 podStartE2EDuration="4.071579429s" totalImagesPullingTime="920.728297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:55:08.070824037 +0000 UTC m=+4145.281647547" watchObservedRunningTime="2026-11-01 06:55:08.071579429 +0000 UTC m=+4145.282402940"
Nov 01 06:55:09.958305 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:09.958270    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:10.750080 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:10.750042    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:11.590485 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:11.590439    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:13.981951 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:13.981921    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"
Nov 01 06:55:14.909244 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:14.909207    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:14.937427 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:14.937402    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7"
Nov 01 06:55:19.761985 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:19.761921    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:23.031913 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:23.031875    5619 generic.go:356] "Generic (PLEG): container finished" podID="e79bdaee-9cd1-4204-8f2f-d8d356851307" containerID="6294debcf630b144d3526c3b4ff89e02ed7c2d16ed2c80c34ffd401b881ec457" exitCode=0
Nov 01 06:55:23.032491 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:23.031919    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b" event={"ID":"e79bdaee-9cd1-4204-8f2f-d8d356851307","Type":"ContainerDied","Data":"6294debcf630b144d3526c3b4ff89e02ed7c2d16ed2c80c34ffd401b881ec457"}
Nov 01 06:55:23.032491 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:23.032409    5619 scope.go:122] "RemoveContainer" containerID="6294debcf630b144d3526c3b4ff89e02ed7c2d16ed2c80c34ffd401b881ec457"
Nov 01 06:55:23.033650 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:23.033617    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:55:23.033762 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:23.033702    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:55:23.033762 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:23.033751    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:55:23.690951 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:23.690915    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:24.038441 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:24.038351    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b" event={"ID":"e79bdaee-9cd1-4204-8f2f-d8d356851307","Type":"ContainerStarted","Data":"bf47b28478e293ff0f54efa624cd81d8207df5f503d0c711b7853cd283243114"}
Nov 01 06:55:24.271544 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:24.271456    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:55:24.272952 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:24.272924    5619 patch_prober.go:29] interesting pod/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 01 06:55:24.277343 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:24.273170    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b" podUID="e79bdaee-9cd1-4204-8f2f-d8d356851307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 01 06:55:24.341226 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:24.341194    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:24.540632 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:24.539475    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:24.915529 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:24.915485    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:25.274815 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:25.274722    5619 patch_prober.go:29] interesting pod/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 01 06:55:25.274815 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:25.274786    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b" podUID="e79bdaee-9cd1-4204-8f2f-d8d356851307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 01 06:55:25.899058 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:55:25.899017    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:55:26.270408 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:26.269767    5619 patch_prober.go:29] interesting pod/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 01 06:55:26.270408 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:26.269836    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b" podUID="e79bdaee-9cd1-4204-8f2f-d8d356851307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 01 06:55:26.894248 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:26.894213    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:55:27.274782 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:27.274688    5619 patch_prober.go:29] interesting pod/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 01 06:55:27.274782 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:27.274742    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b" podUID="e79bdaee-9cd1-4204-8f2f-d8d356851307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 01 06:55:27.734040 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:27.734001    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:28.274795 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:28.274735    5619 patch_prober.go:29] interesting pod/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 01 06:55:28.275176 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:28.274821    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b" podUID="e79bdaee-9cd1-4204-8f2f-d8d356851307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 01 06:55:29.273629 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:29.273597    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:55:29.274667 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:29.274644    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 06:55:29.894750 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:29.894714    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:34.665844 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:34.665811    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:34.811054 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:34.811017    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:35.561763 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:35.561729    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:37.180588 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:37.180550    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:39.568937 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:39.568891    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:40.202297 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:40.202250    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:44.371928 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:44.371892    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:49.170194 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:49.170157    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:53.975476 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:53.975441    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:54.947452 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:54.947410    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:55:55.878524 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:55:55.878479    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:55:58.860493 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:55:58.860457    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:56:03.678445 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:03.678401    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:56:08.471792 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:08.471747    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:56:13.260981 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:13.260942    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:56:18.067917 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:18.067879    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:56:22.879086 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:22.879045    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:56:25.880438 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:56:25.880399    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:56:27.726597 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:27.726565    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:56:32.462264 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:32.462225    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:56:33.431617 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:33.431583    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:56:33.580296 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:33.580266    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:56:36.152198 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:36.152162    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:56:37.275050 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:37.275013    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:56:40.548932 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:40.548892    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:56:42.080743 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:42.080706    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:56:44.175972 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.175933    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bf8b5db8-v4x7w"]
Nov 01 06:56:44.176419 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.176218    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-bf8b5db8-v4x7w" podUID="f577a21b-1c78-4528-a1ea-a6a34c77260c" containerName="manager" containerID="cri-o://f175553e0a74bbc6912196d734d95f9b1111360cee11f348c0548766ad9d1732" gracePeriod=30
Nov 01 06:56:44.435092 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.435036    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bf8b5db8-v4x7w"
Nov 01 06:56:44.445114 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.445089    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f577a21b-1c78-4528-a1ea-a6a34c77260c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f577a21b-1c78-4528-a1ea-a6a34c77260c-metrics-certs\") pod \"f577a21b-1c78-4528-a1ea-a6a34c77260c\" (UID: \"f577a21b-1c78-4528-a1ea-a6a34c77260c\") "
Nov 01 06:56:44.445219 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.445195    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f577a21b-1c78-4528-a1ea-a6a34c77260c-kube-api-access-nkrbf\" (UniqueName: \"kubernetes.io/projected/f577a21b-1c78-4528-a1ea-a6a34c77260c-kube-api-access-nkrbf\") pod \"f577a21b-1c78-4528-a1ea-a6a34c77260c\" (UID: \"f577a21b-1c78-4528-a1ea-a6a34c77260c\") "
Nov 01 06:56:44.445268 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.445223    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f577a21b-1c78-4528-a1ea-a6a34c77260c-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f577a21b-1c78-4528-a1ea-a6a34c77260c-webhook-cert\") pod \"f577a21b-1c78-4528-a1ea-a6a34c77260c\" (UID: \"f577a21b-1c78-4528-a1ea-a6a34c77260c\") "
Nov 01 06:56:44.447236 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.447207    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f577a21b-1c78-4528-a1ea-a6a34c77260c-metrics-certs" pod "f577a21b-1c78-4528-a1ea-a6a34c77260c" (UID: "f577a21b-1c78-4528-a1ea-a6a34c77260c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:56:44.447236 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.447228    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f577a21b-1c78-4528-a1ea-a6a34c77260c-webhook-cert" pod "f577a21b-1c78-4528-a1ea-a6a34c77260c" (UID: "f577a21b-1c78-4528-a1ea-a6a34c77260c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:56:44.447497 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.447479    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f577a21b-1c78-4528-a1ea-a6a34c77260c-kube-api-access-nkrbf" pod "f577a21b-1c78-4528-a1ea-a6a34c77260c" (UID: "f577a21b-1c78-4528-a1ea-a6a34c77260c"). InnerVolumeSpecName "kube-api-access-nkrbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:56:44.459164 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.459136    5619 generic.go:356] "Generic (PLEG): container finished" podID="f577a21b-1c78-4528-a1ea-a6a34c77260c" containerID="f175553e0a74bbc6912196d734d95f9b1111360cee11f348c0548766ad9d1732" exitCode=0
Nov 01 06:56:44.459427 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.459216    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bf8b5db8-v4x7w"
Nov 01 06:56:44.460012 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.459296    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bf8b5db8-v4x7w" event={"ID":"f577a21b-1c78-4528-a1ea-a6a34c77260c","Type":"ContainerDied","Data":"f175553e0a74bbc6912196d734d95f9b1111360cee11f348c0548766ad9d1732"}
Nov 01 06:56:44.460012 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.459668    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bf8b5db8-v4x7w" event={"ID":"f577a21b-1c78-4528-a1ea-a6a34c77260c","Type":"ContainerDied","Data":"e13efecaf5af4cbc98d10ae188d1c67d9c40a245aabed4476cba09d7ca09ce52"}
Nov 01 06:56:44.460012 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.459708    5619 scope.go:122] "RemoveContainer" containerID="f175553e0a74bbc6912196d734d95f9b1111360cee11f348c0548766ad9d1732"
Nov 01 06:56:44.478065 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.478047    5619 scope.go:122] "RemoveContainer" containerID="8f2d8feec7246960a2114cb979591bc391b1173d70f43bde8a0dc8020f432910"
Nov 01 06:56:44.485675 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.485641    5619 scope.go:122] "RemoveContainer" containerID="f175553e0a74bbc6912196d734d95f9b1111360cee11f348c0548766ad9d1732"
Nov 01 06:56:44.485946 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:56:44.485924    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f175553e0a74bbc6912196d734d95f9b1111360cee11f348c0548766ad9d1732\": container with ID starting with f175553e0a74bbc6912196d734d95f9b1111360cee11f348c0548766ad9d1732 not found: ID does not exist" containerID="f175553e0a74bbc6912196d734d95f9b1111360cee11f348c0548766ad9d1732"
Nov 01 06:56:44.486023 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.485954    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f175553e0a74bbc6912196d734d95f9b1111360cee11f348c0548766ad9d1732"} err="failed to get container status \"f175553e0a74bbc6912196d734d95f9b1111360cee11f348c0548766ad9d1732\": rpc error: code = NotFound desc = could not find container \"f175553e0a74bbc6912196d734d95f9b1111360cee11f348c0548766ad9d1732\": container with ID starting with f175553e0a74bbc6912196d734d95f9b1111360cee11f348c0548766ad9d1732 not found: ID does not exist"
Nov 01 06:56:44.486023 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.485971    5619 scope.go:122] "RemoveContainer" containerID="8f2d8feec7246960a2114cb979591bc391b1173d70f43bde8a0dc8020f432910"
Nov 01 06:56:44.486260 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:56:44.486235    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f2d8feec7246960a2114cb979591bc391b1173d70f43bde8a0dc8020f432910\": container with ID starting with 8f2d8feec7246960a2114cb979591bc391b1173d70f43bde8a0dc8020f432910 not found: ID does not exist" containerID="8f2d8feec7246960a2114cb979591bc391b1173d70f43bde8a0dc8020f432910"
Nov 01 06:56:44.486300 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.486271    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f2d8feec7246960a2114cb979591bc391b1173d70f43bde8a0dc8020f432910"} err="failed to get container status \"8f2d8feec7246960a2114cb979591bc391b1173d70f43bde8a0dc8020f432910\": rpc error: code = NotFound desc = could not find container \"8f2d8feec7246960a2114cb979591bc391b1173d70f43bde8a0dc8020f432910\": container with ID starting with 8f2d8feec7246960a2114cb979591bc391b1173d70f43bde8a0dc8020f432910 not found: ID does not exist"
Nov 01 06:56:44.546887 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.546864    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkrbf\" (UniqueName: \"kubernetes.io/projected/f577a21b-1c78-4528-a1ea-a6a34c77260c-kube-api-access-nkrbf\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:56:44.546887 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.546886    5619 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f577a21b-1c78-4528-a1ea-a6a34c77260c-webhook-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:56:44.547068 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.546896    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f577a21b-1c78-4528-a1ea-a6a34c77260c-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:56:44.667943 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.667904    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bf8b5db8-v4x7w"]
Nov 01 06:56:44.671856 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.671826    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bf8b5db8-v4x7w"]
Nov 01 06:56:44.961841 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:44.961809    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f577a21b-1c78-4528-a1ea-a6a34c77260c" path="/var/lib/kubelet/pods/f577a21b-1c78-4528-a1ea-a6a34c77260c/volumes"
Nov 01 06:56:45.509886 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.509856    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bf8b5db8-mnsck"]
Nov 01 06:56:45.510401 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.510384    5619 cpu_manager.go:414] "containerMap: removing container" podUID="f577a21b-1c78-4528-a1ea-a6a34c77260c" containerName="manager"
Nov 01 06:56:45.510482 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.510403    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f577a21b-1c78-4528-a1ea-a6a34c77260c" containerName="manager"
Nov 01 06:56:45.510482 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.510413    5619 cpu_manager.go:414] "containerMap: removing container" podUID="f577a21b-1c78-4528-a1ea-a6a34c77260c" containerName="manager"
Nov 01 06:56:45.510482 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.510419    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f577a21b-1c78-4528-a1ea-a6a34c77260c" containerName="manager"
Nov 01 06:56:45.510595 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.510488    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="f577a21b-1c78-4528-a1ea-a6a34c77260c" containerName="manager"
Nov 01 06:56:45.510595 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.510496    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="f577a21b-1c78-4528-a1ea-a6a34c77260c" containerName="manager"
Nov 01 06:56:45.514631 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.514613    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bf8b5db8-mnsck"
Nov 01 06:56:45.554793 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.554757    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpphf\" (UniqueName: \"kubernetes.io/projected/78fd5fa8-0931-4d8f-8980-abc866d4349e-kube-api-access-tpphf\") pod \"maas-controller-bf8b5db8-mnsck\" (UID: \"78fd5fa8-0931-4d8f-8980-abc866d4349e\") " pod="opendatahub/maas-controller-bf8b5db8-mnsck"
Nov 01 06:56:45.554964 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.554824    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78fd5fa8-0931-4d8f-8980-abc866d4349e-webhook-cert\") pod \"maas-controller-bf8b5db8-mnsck\" (UID: \"78fd5fa8-0931-4d8f-8980-abc866d4349e\") " pod="opendatahub/maas-controller-bf8b5db8-mnsck"
Nov 01 06:56:45.555037 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.554986    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78fd5fa8-0931-4d8f-8980-abc866d4349e-metrics-certs\") pod \"maas-controller-bf8b5db8-mnsck\" (UID: \"78fd5fa8-0931-4d8f-8980-abc866d4349e\") " pod="opendatahub/maas-controller-bf8b5db8-mnsck"
Nov 01 06:56:45.619391 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.619241    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 06:56:45.619391 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.619246    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 06:56:45.619391 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.619318    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 06:56:45.619391 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.619359    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-htpqr\""
Nov 01 06:56:45.621960 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.621933    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bf8b5db8-mnsck"]
Nov 01 06:56:45.655647 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.655608    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpphf\" (UniqueName: \"kubernetes.io/projected/78fd5fa8-0931-4d8f-8980-abc866d4349e-kube-api-access-tpphf\") pod \"maas-controller-bf8b5db8-mnsck\" (UID: \"78fd5fa8-0931-4d8f-8980-abc866d4349e\") " pod="opendatahub/maas-controller-bf8b5db8-mnsck"
Nov 01 06:56:45.655830 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.655674    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78fd5fa8-0931-4d8f-8980-abc866d4349e-webhook-cert\") pod \"maas-controller-bf8b5db8-mnsck\" (UID: \"78fd5fa8-0931-4d8f-8980-abc866d4349e\") " pod="opendatahub/maas-controller-bf8b5db8-mnsck"
Nov 01 06:56:45.655830 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.655752    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78fd5fa8-0931-4d8f-8980-abc866d4349e-metrics-certs\") pod \"maas-controller-bf8b5db8-mnsck\" (UID: \"78fd5fa8-0931-4d8f-8980-abc866d4349e\") " pod="opendatahub/maas-controller-bf8b5db8-mnsck"
Nov 01 06:56:45.658407 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.658382    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78fd5fa8-0931-4d8f-8980-abc866d4349e-metrics-certs\") pod \"maas-controller-bf8b5db8-mnsck\" (UID: \"78fd5fa8-0931-4d8f-8980-abc866d4349e\") " pod="opendatahub/maas-controller-bf8b5db8-mnsck"
Nov 01 06:56:45.658526 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.658490    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78fd5fa8-0931-4d8f-8980-abc866d4349e-webhook-cert\") pod \"maas-controller-bf8b5db8-mnsck\" (UID: \"78fd5fa8-0931-4d8f-8980-abc866d4349e\") " pod="opendatahub/maas-controller-bf8b5db8-mnsck"
Nov 01 06:56:45.717477 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.717445    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpphf\" (UniqueName: \"kubernetes.io/projected/78fd5fa8-0931-4d8f-8980-abc866d4349e-kube-api-access-tpphf\") pod \"maas-controller-bf8b5db8-mnsck\" (UID: \"78fd5fa8-0931-4d8f-8980-abc866d4349e\") " pod="opendatahub/maas-controller-bf8b5db8-mnsck"
Nov 01 06:56:45.827377 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:45.827301    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bf8b5db8-mnsck"
Nov 01 06:56:46.204881 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:46.204756    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bf8b5db8-mnsck"]
Nov 01 06:56:46.207397 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:56:46.207369    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78fd5fa8_0931_4d8f_8980_abc866d4349e.slice/crio-0da0c69c509c1746924e5001eddc1f65909e2c4d80c129ccd5957ef842e778e5 WatchSource:0}: Error finding container 0da0c69c509c1746924e5001eddc1f65909e2c4d80c129ccd5957ef842e778e5: Status 404 returned error can't find the container with id 0da0c69c509c1746924e5001eddc1f65909e2c4d80c129ccd5957ef842e778e5
Nov 01 06:56:46.473969 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:46.473874    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bf8b5db8-mnsck" event={"ID":"78fd5fa8-0931-4d8f-8980-abc866d4349e","Type":"ContainerStarted","Data":"0da0c69c509c1746924e5001eddc1f65909e2c4d80c129ccd5957ef842e778e5"}
Nov 01 06:56:47.481694 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:47.481654    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bf8b5db8-mnsck" event={"ID":"78fd5fa8-0931-4d8f-8980-abc866d4349e","Type":"ContainerStarted","Data":"00535bf614da2efe88c0d3ecc8aca54a9876fe061ba732517ed73cfe16e266d1"}
Nov 01 06:56:47.482225 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:47.481723    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bf8b5db8-mnsck"
Nov 01 06:56:47.601609 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:47.601547    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-bf8b5db8-mnsck" podStartSLOduration=1.5706866640000001 podStartE2EDuration="2.601529094s" totalImagesPullingTime="1.03084243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:56:47.600867276 +0000 UTC m=+4244.811690785" watchObservedRunningTime="2026-11-01 06:56:47.601529094 +0000 UTC m=+4244.812352604"
Nov 01 06:56:54.325055 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:54.325019    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:56:54.467849 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:54.467809    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:56:55.877858 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:56:55.877816    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:56:58.488183 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:56:58.488155    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bf8b5db8-mnsck"
Nov 01 06:57:07.476474 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:07.476437    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:57:12.380832 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:12.380790    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:57:17.181481 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:17.181444    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:57:22.012169 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:22.012125    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:57:24.162415 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.162381    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"]
Nov 01 06:57:24.166317 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.166288    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.224194 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.224165    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac4eb8e8-a937-4744-9e9a-8b8f65803484-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.224360 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.224211    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac4eb8e8-a937-4744-9e9a-8b8f65803484-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.224360 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.224280    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac4eb8e8-a937-4744-9e9a-8b8f65803484-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.224453 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.224370    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac4eb8e8-a937-4744-9e9a-8b8f65803484-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.224453 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.224430    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4eb8e8-a937-4744-9e9a-8b8f65803484-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.224453 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.224447    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm5hm\" (UniqueName: \"kubernetes.io/projected/ac4eb8e8-a937-4744-9e9a-8b8f65803484-kube-api-access-bm5hm\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.224573 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.224465    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac4eb8e8-a937-4744-9e9a-8b8f65803484-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.224614 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.224579    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac4eb8e8-a937-4744-9e9a-8b8f65803484-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.224614 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.224601    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4eb8e8-a937-4744-9e9a-8b8f65803484-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.273060 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.273027    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"]
Nov 01 06:57:24.325282 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.325256    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4eb8e8-a937-4744-9e9a-8b8f65803484-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.325455 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.325289    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm5hm\" (UniqueName: \"kubernetes.io/projected/ac4eb8e8-a937-4744-9e9a-8b8f65803484-kube-api-access-bm5hm\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.325455 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.325308    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac4eb8e8-a937-4744-9e9a-8b8f65803484-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.325455 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.325365    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac4eb8e8-a937-4744-9e9a-8b8f65803484-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.325455 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.325388    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4eb8e8-a937-4744-9e9a-8b8f65803484-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.325687 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.325505    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac4eb8e8-a937-4744-9e9a-8b8f65803484-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.325687 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.325589    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac4eb8e8-a937-4744-9e9a-8b8f65803484-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.325687 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.325642    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac4eb8e8-a937-4744-9e9a-8b8f65803484-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.325687 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.325675    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac4eb8e8-a937-4744-9e9a-8b8f65803484-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.325888 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.325787    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac4eb8e8-a937-4744-9e9a-8b8f65803484-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.325948 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.325899    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4eb8e8-a937-4744-9e9a-8b8f65803484-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.326010 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.325941    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4eb8e8-a937-4744-9e9a-8b8f65803484-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.326148 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.326126    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac4eb8e8-a937-4744-9e9a-8b8f65803484-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.326521 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.326496    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac4eb8e8-a937-4744-9e9a-8b8f65803484-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.327830 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.327807    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac4eb8e8-a937-4744-9e9a-8b8f65803484-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.328117 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.328098    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac4eb8e8-a937-4744-9e9a-8b8f65803484-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.387548 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.387521    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac4eb8e8-a937-4744-9e9a-8b8f65803484-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.387747 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.387723    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm5hm\" (UniqueName: \"kubernetes.io/projected/ac4eb8e8-a937-4744-9e9a-8b8f65803484-kube-api-access-bm5hm\") pod \"maas-default-gateway-openshift-default-855f9574cf-qrvtb\" (UID: \"ac4eb8e8-a937-4744-9e9a-8b8f65803484\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.479348 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.479247    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:24.903945 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.903916    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"]
Nov 01 06:57:24.905435 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:57:24.905404    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4eb8e8_a937_4744_9e9a_8b8f65803484.slice/crio-30987adad026b32146cf65fc85c6c077b311ceef0ab4fc4133f55f494fe298d3 WatchSource:0}: Error finding container 30987adad026b32146cf65fc85c6c077b311ceef0ab4fc4133f55f494fe298d3: Status 404 returned error can't find the container with id 30987adad026b32146cf65fc85c6c077b311ceef0ab4fc4133f55f494fe298d3
Nov 01 06:57:24.908529 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.908487    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:57:24.908657 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.908576    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:57:24.908657 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:24.908621    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 06:57:25.678009 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:25.677972    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb" event={"ID":"ac4eb8e8-a937-4744-9e9a-8b8f65803484","Type":"ContainerStarted","Data":"5ca7e7983201eac3dc7f443f6b9dd2e3c07b9856dbcfe28d4715ff83d8f35dc7"}
Nov 01 06:57:25.678474 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:25.678026    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb" event={"ID":"ac4eb8e8-a937-4744-9e9a-8b8f65803484","Type":"ContainerStarted","Data":"30987adad026b32146cf65fc85c6c077b311ceef0ab4fc4133f55f494fe298d3"}
Nov 01 06:57:25.902304 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:57:25.902265    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:57:25.908625 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:25.908579    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb" podStartSLOduration=1.9085661090000001 podStartE2EDuration="1.908566109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:57:25.857075881 +0000 UTC m=+4283.067899391" watchObservedRunningTime="2026-11-01 06:57:25.908566109 +0000 UTC m=+4283.119389618"
Nov 01 06:57:26.479743 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:26.479702    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:26.481379 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:26.481350    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qrvtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 01 06:57:26.481503 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:26.481403    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb" podUID="ac4eb8e8-a937-4744-9e9a-8b8f65803484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 01 06:57:26.797284 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:26.797182    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:57:27.480583 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:27.480540    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qrvtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 01 06:57:27.480776 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:27.480606    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb" podUID="ac4eb8e8-a937-4744-9e9a-8b8f65803484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 01 06:57:28.480271 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:28.480231    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qrvtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 01 06:57:28.480736 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:28.480293    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb" podUID="ac4eb8e8-a937-4744-9e9a-8b8f65803484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 01 06:57:28.819461 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:28.819369    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:57:28.962347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:28.957413    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bf8b5db8-mnsck"]
Nov 01 06:57:28.962347 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:28.957678    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-bf8b5db8-mnsck" podUID="78fd5fa8-0931-4d8f-8980-abc866d4349e" containerName="manager" containerID="cri-o://00535bf614da2efe88c0d3ecc8aca54a9876fe061ba732517ed73cfe16e266d1" gracePeriod=30
Nov 01 06:57:29.232139 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:29.232114    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bf8b5db8-mnsck"
Nov 01 06:57:29.299675 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:29.299646    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78fd5fa8-0931-4d8f-8980-abc866d4349e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/78fd5fa8-0931-4d8f-8980-abc866d4349e-metrics-certs\") pod \"78fd5fa8-0931-4d8f-8980-abc866d4349e\" (UID: \"78fd5fa8-0931-4d8f-8980-abc866d4349e\") "
Nov 01 06:57:29.299851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:29.299743    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78fd5fa8-0931-4d8f-8980-abc866d4349e-kube-api-access-tpphf\" (UniqueName: \"kubernetes.io/projected/78fd5fa8-0931-4d8f-8980-abc866d4349e-kube-api-access-tpphf\") pod \"78fd5fa8-0931-4d8f-8980-abc866d4349e\" (UID: \"78fd5fa8-0931-4d8f-8980-abc866d4349e\") "
Nov 01 06:57:29.299851 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:29.299797    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78fd5fa8-0931-4d8f-8980-abc866d4349e-webhook-cert\" (UniqueName: \"kubernetes.io/secret/78fd5fa8-0931-4d8f-8980-abc866d4349e-webhook-cert\") pod \"78fd5fa8-0931-4d8f-8980-abc866d4349e\" (UID: \"78fd5fa8-0931-4d8f-8980-abc866d4349e\") "
Nov 01 06:57:29.301847 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:29.301813    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78fd5fa8-0931-4d8f-8980-abc866d4349e-metrics-certs" pod "78fd5fa8-0931-4d8f-8980-abc866d4349e" (UID: "78fd5fa8-0931-4d8f-8980-abc866d4349e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:57:29.301847 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:29.301830    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78fd5fa8-0931-4d8f-8980-abc866d4349e-webhook-cert" pod "78fd5fa8-0931-4d8f-8980-abc866d4349e" (UID: "78fd5fa8-0931-4d8f-8980-abc866d4349e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 06:57:29.302016 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:29.301863    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78fd5fa8-0931-4d8f-8980-abc866d4349e-kube-api-access-tpphf" pod "78fd5fa8-0931-4d8f-8980-abc866d4349e" (UID: "78fd5fa8-0931-4d8f-8980-abc866d4349e"). InnerVolumeSpecName "kube-api-access-tpphf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:57:29.401176 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:29.401141    5619 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78fd5fa8-0931-4d8f-8980-abc866d4349e-webhook-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:57:29.401176 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:29.401177    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78fd5fa8-0931-4d8f-8980-abc866d4349e-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:57:29.401408 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:29.401192    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpphf\" (UniqueName: \"kubernetes.io/projected/78fd5fa8-0931-4d8f-8980-abc866d4349e-kube-api-access-tpphf\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:57:29.479728 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:29.479664    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qrvtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 01 06:57:29.479728 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:29.479723    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb" podUID="ac4eb8e8-a937-4744-9e9a-8b8f65803484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 01 06:57:29.697829 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:29.697734    5619 generic.go:356] "Generic (PLEG): container finished" podID="78fd5fa8-0931-4d8f-8980-abc866d4349e" containerID="00535bf614da2efe88c0d3ecc8aca54a9876fe061ba732517ed73cfe16e266d1" exitCode=0
Nov 01 06:57:29.697829 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:29.697783    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bf8b5db8-mnsck"
Nov 01 06:57:29.697829 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:29.697814    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bf8b5db8-mnsck" event={"ID":"78fd5fa8-0931-4d8f-8980-abc866d4349e","Type":"ContainerDied","Data":"00535bf614da2efe88c0d3ecc8aca54a9876fe061ba732517ed73cfe16e266d1"}
Nov 01 06:57:29.698389 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:29.697873    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bf8b5db8-mnsck" event={"ID":"78fd5fa8-0931-4d8f-8980-abc866d4349e","Type":"ContainerDied","Data":"0da0c69c509c1746924e5001eddc1f65909e2c4d80c129ccd5957ef842e778e5"}
Nov 01 06:57:29.698389 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:29.697903    5619 scope.go:122] "RemoveContainer" containerID="00535bf614da2efe88c0d3ecc8aca54a9876fe061ba732517ed73cfe16e266d1"
Nov 01 06:57:29.707813 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:29.707794    5619 scope.go:122] "RemoveContainer" containerID="00535bf614da2efe88c0d3ecc8aca54a9876fe061ba732517ed73cfe16e266d1"
Nov 01 06:57:29.708104 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:57:29.708055    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00535bf614da2efe88c0d3ecc8aca54a9876fe061ba732517ed73cfe16e266d1\": container with ID starting with 00535bf614da2efe88c0d3ecc8aca54a9876fe061ba732517ed73cfe16e266d1 not found: ID does not exist" containerID="00535bf614da2efe88c0d3ecc8aca54a9876fe061ba732517ed73cfe16e266d1"
Nov 01 06:57:29.708208 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:29.708109    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00535bf614da2efe88c0d3ecc8aca54a9876fe061ba732517ed73cfe16e266d1"} err="failed to get container status \"00535bf614da2efe88c0d3ecc8aca54a9876fe061ba732517ed73cfe16e266d1\": rpc error: code = NotFound desc = could not find container \"00535bf614da2efe88c0d3ecc8aca54a9876fe061ba732517ed73cfe16e266d1\": container with ID starting with 00535bf614da2efe88c0d3ecc8aca54a9876fe061ba732517ed73cfe16e266d1 not found: ID does not exist"
Nov 01 06:57:29.882857 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:29.882819    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bf8b5db8-mnsck"]
Nov 01 06:57:29.886885 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:29.886860    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bf8b5db8-mnsck"]
Nov 01 06:57:30.480181 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:30.480145    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qrvtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 01 06:57:30.480393 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:30.480213    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb" podUID="ac4eb8e8-a937-4744-9e9a-8b8f65803484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 01 06:57:30.969908 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:30.969868    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78fd5fa8-0931-4d8f-8980-abc866d4349e" path="/var/lib/kubelet/pods/78fd5fa8-0931-4d8f-8980-abc866d4349e/volumes"
Nov 01 06:57:31.479668 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:31.479629    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qrvtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 01 06:57:31.479849 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:31.479697    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb" podUID="ac4eb8e8-a937-4744-9e9a-8b8f65803484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 01 06:57:33.160368 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:33.160317    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:33.160758 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:33.160647    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:33.161432 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:33.161415    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb"
Nov 01 06:57:36.506495 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:36.506460    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:57:40.897484 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:40.897447    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:57:41.151379 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:41.151265    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:57:41.303155 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:41.303118    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bf8b5db8-w9m7j"]
Nov 01 06:57:41.303601 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:41.303587    5619 cpu_manager.go:414] "containerMap: removing container" podUID="78fd5fa8-0931-4d8f-8980-abc866d4349e" containerName="manager"
Nov 01 06:57:41.303654 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:41.303605    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78fd5fa8-0931-4d8f-8980-abc866d4349e" containerName="manager"
Nov 01 06:57:41.303691 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:41.303665    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="78fd5fa8-0931-4d8f-8980-abc866d4349e" containerName="manager"
Nov 01 06:57:41.306742 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:41.306726    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bf8b5db8-w9m7j"
Nov 01 06:57:41.411171 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:41.411051    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 06:57:41.411171 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:41.411051    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-htpqr\""
Nov 01 06:57:41.411171 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:41.411102    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 06:57:41.411171 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:41.411122    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 06:57:41.412943 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:41.412920    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bf8b5db8-w9m7j"]
Nov 01 06:57:41.421766 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:41.421743    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a322e32c-aaf3-448f-8fbd-49459435c6b5-webhook-cert\") pod \"maas-controller-bf8b5db8-w9m7j\" (UID: \"a322e32c-aaf3-448f-8fbd-49459435c6b5\") " pod="opendatahub/maas-controller-bf8b5db8-w9m7j"
Nov 01 06:57:41.421885 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:41.421868    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a322e32c-aaf3-448f-8fbd-49459435c6b5-metrics-certs\") pod \"maas-controller-bf8b5db8-w9m7j\" (UID: \"a322e32c-aaf3-448f-8fbd-49459435c6b5\") " pod="opendatahub/maas-controller-bf8b5db8-w9m7j"
Nov 01 06:57:41.421929 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:41.421918    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r2s9\" (UniqueName: \"kubernetes.io/projected/a322e32c-aaf3-448f-8fbd-49459435c6b5-kube-api-access-4r2s9\") pod \"maas-controller-bf8b5db8-w9m7j\" (UID: \"a322e32c-aaf3-448f-8fbd-49459435c6b5\") " pod="opendatahub/maas-controller-bf8b5db8-w9m7j"
Nov 01 06:57:41.523075 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:41.523036    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a322e32c-aaf3-448f-8fbd-49459435c6b5-metrics-certs\") pod \"maas-controller-bf8b5db8-w9m7j\" (UID: \"a322e32c-aaf3-448f-8fbd-49459435c6b5\") " pod="opendatahub/maas-controller-bf8b5db8-w9m7j"
Nov 01 06:57:41.523230 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:41.523086    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r2s9\" (UniqueName: \"kubernetes.io/projected/a322e32c-aaf3-448f-8fbd-49459435c6b5-kube-api-access-4r2s9\") pod \"maas-controller-bf8b5db8-w9m7j\" (UID: \"a322e32c-aaf3-448f-8fbd-49459435c6b5\") " pod="opendatahub/maas-controller-bf8b5db8-w9m7j"
Nov 01 06:57:41.523230 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:41.523153    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a322e32c-aaf3-448f-8fbd-49459435c6b5-webhook-cert\") pod \"maas-controller-bf8b5db8-w9m7j\" (UID: \"a322e32c-aaf3-448f-8fbd-49459435c6b5\") " pod="opendatahub/maas-controller-bf8b5db8-w9m7j"
Nov 01 06:57:41.525610 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:41.525583    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a322e32c-aaf3-448f-8fbd-49459435c6b5-metrics-certs\") pod \"maas-controller-bf8b5db8-w9m7j\" (UID: \"a322e32c-aaf3-448f-8fbd-49459435c6b5\") " pod="opendatahub/maas-controller-bf8b5db8-w9m7j"
Nov 01 06:57:41.525721 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:41.525613    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a322e32c-aaf3-448f-8fbd-49459435c6b5-webhook-cert\") pod \"maas-controller-bf8b5db8-w9m7j\" (UID: \"a322e32c-aaf3-448f-8fbd-49459435c6b5\") " pod="opendatahub/maas-controller-bf8b5db8-w9m7j"
Nov 01 06:57:41.585529 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:41.585495    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r2s9\" (UniqueName: \"kubernetes.io/projected/a322e32c-aaf3-448f-8fbd-49459435c6b5-kube-api-access-4r2s9\") pod \"maas-controller-bf8b5db8-w9m7j\" (UID: \"a322e32c-aaf3-448f-8fbd-49459435c6b5\") " pod="opendatahub/maas-controller-bf8b5db8-w9m7j"
Nov 01 06:57:41.619358 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:41.619302    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bf8b5db8-w9m7j"
Nov 01 06:57:42.044304 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:42.044251    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bf8b5db8-w9m7j"]
Nov 01 06:57:42.046573 ip-10-0-129-213 kubenswrapper[5619]: W1101 06:57:42.046546    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda322e32c_aaf3_448f_8fbd_49459435c6b5.slice/crio-28032f65d2406f7e8f6996a94cba6db1c9118ba4a6c4388559164c11eb7a2c2b WatchSource:0}: Error finding container 28032f65d2406f7e8f6996a94cba6db1c9118ba4a6c4388559164c11eb7a2c2b: Status 404 returned error can't find the container with id 28032f65d2406f7e8f6996a94cba6db1c9118ba4a6c4388559164c11eb7a2c2b
Nov 01 06:57:42.763197 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:42.763163    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bf8b5db8-w9m7j" event={"ID":"a322e32c-aaf3-448f-8fbd-49459435c6b5","Type":"ContainerStarted","Data":"28032f65d2406f7e8f6996a94cba6db1c9118ba4a6c4388559164c11eb7a2c2b"}
Nov 01 06:57:43.769279 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:43.769241    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bf8b5db8-w9m7j" event={"ID":"a322e32c-aaf3-448f-8fbd-49459435c6b5","Type":"ContainerStarted","Data":"82cff9f7fc3be894349f9f2670499e9b2863fe6174ca2dbaa8facb9e6b217f56"}
Nov 01 06:57:43.769745 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:43.769315    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bf8b5db8-w9m7j"
Nov 01 06:57:43.889508 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:43.889448    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-bf8b5db8-w9m7j" podStartSLOduration=2.085973036 podStartE2EDuration="2.889432319s" totalImagesPullingTime="803.459283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:57:43.888659577 +0000 UTC m=+4301.099483095" watchObservedRunningTime="2026-11-01 06:57:43.889432319 +0000 UTC m=+4301.100255827"
Nov 01 06:57:43.965697 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:43.965666    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:57:46.265759 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:46.265724    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:57:54.776941 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:57:54.776910    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bf8b5db8-w9m7j"
Nov 01 06:57:55.879973 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:57:55.879926    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:58:07.666839 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:07.666799    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:58:12.374781 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:12.374689    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:58:17.161615 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:17.161580    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:58:21.852721 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:21.852685    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:58:25.879146 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:58:25.879110    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:58:26.641255 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:26.641220    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:58:27.699290 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:27.699254    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q"]
Nov 01 06:58:27.699794 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:27.699502    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q" podUID="b3857be3-6a36-43f7-adec-7c924b444dd1" containerName="manager" containerID="cri-o://ef8e560d23db763b749620081c987357bb4d4c8af10f60e29bfa64112e0a7990" gracePeriod=10
Nov 01 06:58:27.954490 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:27.954426    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q"
Nov 01 06:58:27.984990 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:27.984958    5619 generic.go:356] "Generic (PLEG): container finished" podID="b3857be3-6a36-43f7-adec-7c924b444dd1" containerID="ef8e560d23db763b749620081c987357bb4d4c8af10f60e29bfa64112e0a7990" exitCode=0
Nov 01 06:58:27.985117 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:27.984998    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q"
Nov 01 06:58:27.985117 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:27.985031    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q" event={"ID":"b3857be3-6a36-43f7-adec-7c924b444dd1","Type":"ContainerDied","Data":"ef8e560d23db763b749620081c987357bb4d4c8af10f60e29bfa64112e0a7990"}
Nov 01 06:58:27.985117 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:27.985075    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q" event={"ID":"b3857be3-6a36-43f7-adec-7c924b444dd1","Type":"ContainerDied","Data":"24f7672e0e7e55007db4d97c89159a83fbbbac0b1ab664538816cd5dd553b89b"}
Nov 01 06:58:27.985117 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:27.985098    5619 scope.go:122] "RemoveContainer" containerID="ef8e560d23db763b749620081c987357bb4d4c8af10f60e29bfa64112e0a7990"
Nov 01 06:58:27.998985 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:27.998961    5619 scope.go:122] "RemoveContainer" containerID="ef8e560d23db763b749620081c987357bb4d4c8af10f60e29bfa64112e0a7990"
Nov 01 06:58:27.999260 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:58:27.999239    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef8e560d23db763b749620081c987357bb4d4c8af10f60e29bfa64112e0a7990\": container with ID starting with ef8e560d23db763b749620081c987357bb4d4c8af10f60e29bfa64112e0a7990 not found: ID does not exist" containerID="ef8e560d23db763b749620081c987357bb4d4c8af10f60e29bfa64112e0a7990"
Nov 01 06:58:27.999367 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:27.999266    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8e560d23db763b749620081c987357bb4d4c8af10f60e29bfa64112e0a7990"} err="failed to get container status \"ef8e560d23db763b749620081c987357bb4d4c8af10f60e29bfa64112e0a7990\": rpc error: code = NotFound desc = could not find container \"ef8e560d23db763b749620081c987357bb4d4c8af10f60e29bfa64112e0a7990\": container with ID starting with ef8e560d23db763b749620081c987357bb4d4c8af10f60e29bfa64112e0a7990 not found: ID does not exist"
Nov 01 06:58:28.096917 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:28.096880    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3857be3-6a36-43f7-adec-7c924b444dd1-kube-api-access-mvmr4\" (UniqueName: \"kubernetes.io/projected/b3857be3-6a36-43f7-adec-7c924b444dd1-kube-api-access-mvmr4\") pod \"b3857be3-6a36-43f7-adec-7c924b444dd1\" (UID: \"b3857be3-6a36-43f7-adec-7c924b444dd1\") "
Nov 01 06:58:28.097107 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:28.097026    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3857be3-6a36-43f7-adec-7c924b444dd1-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3857be3-6a36-43f7-adec-7c924b444dd1-extensions-socket-volume\") pod \"b3857be3-6a36-43f7-adec-7c924b444dd1\" (UID: \"b3857be3-6a36-43f7-adec-7c924b444dd1\") "
Nov 01 06:58:28.097417 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:28.097394    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3857be3-6a36-43f7-adec-7c924b444dd1-extensions-socket-volume" pod "b3857be3-6a36-43f7-adec-7c924b444dd1" (UID: "b3857be3-6a36-43f7-adec-7c924b444dd1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 06:58:28.097549 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:28.097536    5619 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3857be3-6a36-43f7-adec-7c924b444dd1-extensions-socket-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:58:28.103522 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:28.103491    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3857be3-6a36-43f7-adec-7c924b444dd1-kube-api-access-mvmr4" pod "b3857be3-6a36-43f7-adec-7c924b444dd1" (UID: "b3857be3-6a36-43f7-adec-7c924b444dd1"). InnerVolumeSpecName "kube-api-access-mvmr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 06:58:28.198902 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:28.198869    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvmr4\" (UniqueName: \"kubernetes.io/projected/b3857be3-6a36-43f7-adec-7c924b444dd1-kube-api-access-mvmr4\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 06:58:28.409791 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:28.409756    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q"]
Nov 01 06:58:28.413451 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:28.413428    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zxf6q"]
Nov 01 06:58:28.961758 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:28.961724    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3857be3-6a36-43f7-adec-7c924b444dd1" path="/var/lib/kubelet/pods/b3857be3-6a36-43f7-adec-7c924b444dd1/volumes"
Nov 01 06:58:31.362226 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:31.362186    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:58:36.168447 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:36.168407    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:58:36.846061 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:36.846027    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:58:40.861460 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:40.861417    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:58:41.238416 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:41.238311    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:58:45.650508 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:45.650470    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:58:50.359022 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:50.358994    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:58:55.153798 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:55.153761    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:58:55.877756 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:58:55.877723    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:58:59.852211 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:58:59.852171    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:59:04.652798 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:04.652760    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:59:09.086519 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:09.086481    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:59:09.273248 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:09.273209    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:59:09.414601 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:09.414565    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:59:14.150002 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:14.149959    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:59:18.856012 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:18.855973    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:59:23.646257 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:23.646216    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:59:25.879913 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:59:25.879871    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:59:28.355502 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:28.355466    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:59:31.431905 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:31.431870    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hw2bp"]
Nov 01 06:59:31.432403 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:31.432278    5619 cpu_manager.go:414] "containerMap: removing container" podUID="b3857be3-6a36-43f7-adec-7c924b444dd1" containerName="manager"
Nov 01 06:59:31.432403 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:31.432289    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3857be3-6a36-43f7-adec-7c924b444dd1" containerName="manager"
Nov 01 06:59:31.432403 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:31.432375    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3857be3-6a36-43f7-adec-7c924b444dd1" containerName="manager"
Nov 01 06:59:31.435816 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:31.435792    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hw2bp"
Nov 01 06:59:31.491019 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:31.490985    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hw2bp"]
Nov 01 06:59:31.512950 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:31.512921    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nc8x\" (UniqueName: \"kubernetes.io/projected/2a0527b8-6adb-4ed9-a183-8263f8cbbb0d-kube-api-access-5nc8x\") pod \"kuadrant-operator-controller-manager-678d87766f-hw2bp\" (UID: \"2a0527b8-6adb-4ed9-a183-8263f8cbbb0d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hw2bp"
Nov 01 06:59:31.513082 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:31.513047    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a0527b8-6adb-4ed9-a183-8263f8cbbb0d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hw2bp\" (UID: \"2a0527b8-6adb-4ed9-a183-8263f8cbbb0d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hw2bp"
Nov 01 06:59:31.540797 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:31.540773    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 06:59:31.540797 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:31.540785    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 06:59:31.540978 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:31.540812    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vhvbn\""
Nov 01 06:59:31.614426 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:31.614381    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a0527b8-6adb-4ed9-a183-8263f8cbbb0d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hw2bp\" (UID: \"2a0527b8-6adb-4ed9-a183-8263f8cbbb0d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hw2bp"
Nov 01 06:59:31.614627 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:31.614523    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nc8x\" (UniqueName: \"kubernetes.io/projected/2a0527b8-6adb-4ed9-a183-8263f8cbbb0d-kube-api-access-5nc8x\") pod \"kuadrant-operator-controller-manager-678d87766f-hw2bp\" (UID: \"2a0527b8-6adb-4ed9-a183-8263f8cbbb0d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hw2bp"
Nov 01 06:59:31.614813 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:31.614794    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a0527b8-6adb-4ed9-a183-8263f8cbbb0d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hw2bp\" (UID: \"2a0527b8-6adb-4ed9-a183-8263f8cbbb0d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hw2bp"
Nov 01 06:59:31.683184 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:31.683112    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nc8x\" (UniqueName: \"kubernetes.io/projected/2a0527b8-6adb-4ed9-a183-8263f8cbbb0d-kube-api-access-5nc8x\") pod \"kuadrant-operator-controller-manager-678d87766f-hw2bp\" (UID: \"2a0527b8-6adb-4ed9-a183-8263f8cbbb0d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hw2bp"
Nov 01 06:59:31.748244 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:31.748211    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hw2bp"
Nov 01 06:59:32.227796 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:32.225525    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hw2bp"]
Nov 01 06:59:32.297560 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:32.297533    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hw2bp" event={"ID":"2a0527b8-6adb-4ed9-a183-8263f8cbbb0d","Type":"ContainerStarted","Data":"064c412b86c884df359de3b93e81054235b16badc923ae741ff68a867e8eec02"}
Nov 01 06:59:32.297694 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:32.297582    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hw2bp" event={"ID":"2a0527b8-6adb-4ed9-a183-8263f8cbbb0d","Type":"ContainerStarted","Data":"09bfe989243a31e02784449a204ee9db86992b0f06a6121a3cabd68cceeefd10"}
Nov 01 06:59:32.297694 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:32.297682    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hw2bp"
Nov 01 06:59:32.420492 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:32.420438    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hw2bp" podStartSLOduration=1.420426388 podStartE2EDuration="1.420426388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:59:32.41888648 +0000 UTC m=+4409.629710000" watchObservedRunningTime="2026-11-01 06:59:32.420426388 +0000 UTC m=+4409.631249897"
Nov 01 06:59:33.166666 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:33.166633    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:59:37.194372 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:37.194310    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:59:37.946800 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:37.946768    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:59:41.586158 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:41.586060    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:59:42.679495 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:42.679448    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:59:43.304623 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:43.304592    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hw2bp"
Nov 01 06:59:47.544106 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:47.544068    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:59:48.264263 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:48.264227    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:59:52.347835 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:52.347796    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 06:59:55.879964 ip-10-0-129-213 kubenswrapper[5619]: E1101 06:59:55.879922    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 06:59:57.057897 ip-10-0-129-213 kubenswrapper[5619]: I1101 06:59:57.057859    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:00:01.863945 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:01.863911    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:00:06.647385 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:06.647339    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:00:11.461602 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:11.461564    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:00:16.173726 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:16.173687    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:00:20.969989 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:20.969913    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:00:24.175074 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.175039    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"]
Nov 01 07:00:24.178673 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.178651    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.247475 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.247444    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f706dfe8-5cf7-42c5-855e-85f021da7b96-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.247616 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.247506    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdprp\" (UniqueName: \"kubernetes.io/projected/f706dfe8-5cf7-42c5-855e-85f021da7b96-kube-api-access-xdprp\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.247616 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.247571    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f706dfe8-5cf7-42c5-855e-85f021da7b96-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.247616 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.247609    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f706dfe8-5cf7-42c5-855e-85f021da7b96-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.247749 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.247702    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f706dfe8-5cf7-42c5-855e-85f021da7b96-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.247749 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.247728    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f706dfe8-5cf7-42c5-855e-85f021da7b96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.247872 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.247774    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f706dfe8-5cf7-42c5-855e-85f021da7b96-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.247872 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.247848    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f706dfe8-5cf7-42c5-855e-85f021da7b96-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.247872 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.247868    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f706dfe8-5cf7-42c5-855e-85f021da7b96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.286440 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.286408    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"]
Nov 01 07:00:24.349148 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.349117    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f706dfe8-5cf7-42c5-855e-85f021da7b96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.349362 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.349174    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f706dfe8-5cf7-42c5-855e-85f021da7b96-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.349362 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.349202    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f706dfe8-5cf7-42c5-855e-85f021da7b96-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.349362 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.349223    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f706dfe8-5cf7-42c5-855e-85f021da7b96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.349362 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.349257    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f706dfe8-5cf7-42c5-855e-85f021da7b96-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.349362 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.349310    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdprp\" (UniqueName: \"kubernetes.io/projected/f706dfe8-5cf7-42c5-855e-85f021da7b96-kube-api-access-xdprp\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.349362 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.349354    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f706dfe8-5cf7-42c5-855e-85f021da7b96-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.349729 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.349392    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f706dfe8-5cf7-42c5-855e-85f021da7b96-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.349729 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.349455    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f706dfe8-5cf7-42c5-855e-85f021da7b96-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.349729 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.349673    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f706dfe8-5cf7-42c5-855e-85f021da7b96-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.349906 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.349822    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f706dfe8-5cf7-42c5-855e-85f021da7b96-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.349966 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.349899    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f706dfe8-5cf7-42c5-855e-85f021da7b96-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.350028 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.350002    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f706dfe8-5cf7-42c5-855e-85f021da7b96-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.350358 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.350342    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f706dfe8-5cf7-42c5-855e-85f021da7b96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.351664 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.351644    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f706dfe8-5cf7-42c5-855e-85f021da7b96-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.351794 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.351771    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f706dfe8-5cf7-42c5-855e-85f021da7b96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.411906 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.411870    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f706dfe8-5cf7-42c5-855e-85f021da7b96-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.412054 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.412031    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdprp\" (UniqueName: \"kubernetes.io/projected/f706dfe8-5cf7-42c5-855e-85f021da7b96-kube-api-access-xdprp\") pod \"maas-default-gateway-openshift-default-855f9574cf-6sjbq\" (UID: \"f706dfe8-5cf7-42c5-855e-85f021da7b96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.492563 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.492473    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:24.919112 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.919080    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"]
Nov 01 07:00:24.921054 ip-10-0-129-213 kubenswrapper[5619]: W1101 07:00:24.921022    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf706dfe8_5cf7_42c5_855e_85f021da7b96.slice/crio-3d1e44b8c01475462233e02d222646be88c8966091ab205f19e76d6e5b14a7d8 WatchSource:0}: Error finding container 3d1e44b8c01475462233e02d222646be88c8966091ab205f19e76d6e5b14a7d8: Status 404 returned error can't find the container with id 3d1e44b8c01475462233e02d222646be88c8966091ab205f19e76d6e5b14a7d8
Nov 01 07:00:24.923373 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.923342    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 07:00:24.923458 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.923412    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 07:00:24.923458 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:24.923441    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 07:00:25.584665 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:25.584624    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq" event={"ID":"f706dfe8-5cf7-42c5-855e-85f021da7b96","Type":"ContainerStarted","Data":"eb01131909c5b61374a8f519d7e0aa90f873af8cc3d9eafb27fc2d49a84e50ff"}
Nov 01 07:00:25.585091 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:25.584694    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq" event={"ID":"f706dfe8-5cf7-42c5-855e-85f021da7b96","Type":"ContainerStarted","Data":"3d1e44b8c01475462233e02d222646be88c8966091ab205f19e76d6e5b14a7d8"}
Nov 01 07:00:25.681888 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:25.681853    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:00:25.707429 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:25.707378    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq" podStartSLOduration=1.707363247 podStartE2EDuration="1.707363247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:25.705743405 +0000 UTC m=+4462.916566916" watchObservedRunningTime="2026-11-01 07:00:25.707363247 +0000 UTC m=+4462.918186756"
Nov 01 07:00:25.900717 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:00:25.900679    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:00:26.492927 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:26.492885    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:26.494344 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:26.494301    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6sjbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 01 07:00:26.494447 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:26.494372    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq" podUID="f706dfe8-5cf7-42c5-855e-85f021da7b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 01 07:00:27.493693 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:27.493659    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6sjbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 01 07:00:27.494094 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:27.493712    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq" podUID="f706dfe8-5cf7-42c5-855e-85f021da7b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 01 07:00:28.493752 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:28.493709    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6sjbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 01 07:00:28.494260 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:28.493777    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq" podUID="f706dfe8-5cf7-42c5-855e-85f021da7b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 01 07:00:29.493711 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:29.493671    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6sjbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 01 07:00:29.493920 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:29.493747    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq" podUID="f706dfe8-5cf7-42c5-855e-85f021da7b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 01 07:00:30.493737 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:30.493702    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6sjbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 01 07:00:30.493912 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:30.493757    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq" podUID="f706dfe8-5cf7-42c5-855e-85f021da7b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 01 07:00:30.597412 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:30.597367    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:00:31.493414 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:31.493378    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6sjbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 01 07:00:31.493628 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:31.493437    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq" podUID="f706dfe8-5cf7-42c5-855e-85f021da7b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 01 07:00:33.189177 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:33.189143    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:33.189806 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:33.189650    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:33.190416 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:33.190398    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq"
Nov 01 07:00:35.364243 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:35.364202    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:00:37.551557 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:37.551521    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:00:40.163559 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:40.163517    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:00:41.991018 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:41.990979    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:00:44.978713 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:44.978680    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:00:49.922348 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:49.921899    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:00:54.769176 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:54.769134    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:00:55.880212 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:00:55.880178    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:00:56.301007 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:56.300910    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:00:59.560890 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:00:59.560858    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:01:04.268160 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:01:04.268109    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:01:09.082623 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:01:09.082584    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:01:13.871023 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:01:13.870931    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:01:17.573207 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:01:17.573170    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:01:17.767122 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:01:17.767085    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:01:18.668620 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:01:18.668582    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:01:23.605966 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:01:23.605927    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:01:25.879580 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:01:25.879546    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:01:28.522508 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:01:28.522466    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:01:33.163623 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:01:33.163586    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:01:37.899033 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:01:37.898995    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:01:38.091092 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:01:38.091054    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:01:42.415347 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:01:42.414940    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:01:42.839507 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:01:42.839405    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:01:47.659610 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:01:47.659560    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:01:52.478604 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:01:52.478563    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:01:55.878688 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:01:55.878638    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:01:57.366032 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:01:57.365994    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:02:02.088449 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:02:02.088413    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:02:06.967423 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:02:06.967385    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:02:11.674705 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:02:11.674667    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:02:16.711553 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:02:16.711497    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:02:21.460876 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:02:21.460834    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:02:25.936728 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:02:25.936661    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:02:26.307287 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:02:26.307052    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:02:31.147301 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:02:31.147261    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:02:35.752782 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:02:35.752743    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:02:35.958892 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:02:35.958855    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:02:38.248856 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:02:38.248826    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:02:40.674771 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:02:40.674734    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:02:42.743963 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:02:42.743928    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:02:43.694621 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:02:43.694578    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:02:44.278577 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:02:44.278544    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:02:44.476072 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:02:44.476037    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:02:45.469428 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:02:45.469376    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:02:50.274478 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:02:50.274443    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:02:55.070389 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:02:55.070354    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:02:55.878772 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:02:55.878737    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:02:59.872386 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:02:59.872350    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:03:04.682165 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:04.682124    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:03:09.525511 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:09.525448    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:03:14.269108 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:14.269068    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:03:17.299697 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:17.299658    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:03:19.136167 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:19.136124    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:03:22.371901 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:22.371856    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:03:23.970883 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:23.970844    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:03:24.318484 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:24.318396    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:03:25.878783 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:03:25.878736    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:03:28.667133 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:28.667094    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:03:29.477780 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:29.477749    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7"]
Nov 01 07:03:29.831996 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:29.831873    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:03:29.975977 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:29.975932    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:03:30.687915 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:30.687869    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:03:34.478371 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:34.478293    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7" podUID="7a56ff1b-d00d-4595-a328-a859dc31b021" containerName="payload-processing" containerID="cri-o://0103b28ff94b1170327f28848747d094c3d9d2675920823e8e112dc185fcaf8d" gracePeriod=25
Nov 01 07:03:34.723419 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:34.723392    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7"
Nov 01 07:03:34.750628 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:34.750565    5619 generic.go:356] "Generic (PLEG): container finished" podID="7a56ff1b-d00d-4595-a328-a859dc31b021" containerID="0103b28ff94b1170327f28848747d094c3d9d2675920823e8e112dc185fcaf8d" exitCode=0
Nov 01 07:03:34.750628 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:34.750607    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7"
Nov 01 07:03:34.750803 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:34.750627    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7" event={"ID":"7a56ff1b-d00d-4595-a328-a859dc31b021","Type":"ContainerDied","Data":"0103b28ff94b1170327f28848747d094c3d9d2675920823e8e112dc185fcaf8d"}
Nov 01 07:03:34.750803 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:34.750671    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7" event={"ID":"7a56ff1b-d00d-4595-a328-a859dc31b021","Type":"ContainerDied","Data":"c6efa0a77666837821831cc5e809abfad85c293d71a9b718f1ccbcae45cd094a"}
Nov 01 07:03:34.750803 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:34.750693    5619 scope.go:122] "RemoveContainer" containerID="0103b28ff94b1170327f28848747d094c3d9d2675920823e8e112dc185fcaf8d"
Nov 01 07:03:34.759683 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:34.759667    5619 scope.go:122] "RemoveContainer" containerID="0103b28ff94b1170327f28848747d094c3d9d2675920823e8e112dc185fcaf8d"
Nov 01 07:03:34.759950 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:03:34.759927    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0103b28ff94b1170327f28848747d094c3d9d2675920823e8e112dc185fcaf8d\": container with ID starting with 0103b28ff94b1170327f28848747d094c3d9d2675920823e8e112dc185fcaf8d not found: ID does not exist" containerID="0103b28ff94b1170327f28848747d094c3d9d2675920823e8e112dc185fcaf8d"
Nov 01 07:03:34.760018 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:34.759962    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0103b28ff94b1170327f28848747d094c3d9d2675920823e8e112dc185fcaf8d"} err="failed to get container status \"0103b28ff94b1170327f28848747d094c3d9d2675920823e8e112dc185fcaf8d\": rpc error: code = NotFound desc = could not find container \"0103b28ff94b1170327f28848747d094c3d9d2675920823e8e112dc185fcaf8d\": container with ID starting with 0103b28ff94b1170327f28848747d094c3d9d2675920823e8e112dc185fcaf8d not found: ID does not exist"
Nov 01 07:03:34.823864 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:34.823844    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a56ff1b-d00d-4595-a328-a859dc31b021-kube-api-access-z2ccc\" (UniqueName: \"kubernetes.io/projected/7a56ff1b-d00d-4595-a328-a859dc31b021-kube-api-access-z2ccc\") pod \"7a56ff1b-d00d-4595-a328-a859dc31b021\" (UID: \"7a56ff1b-d00d-4595-a328-a859dc31b021\") "
Nov 01 07:03:34.823980 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:34.823895    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a56ff1b-d00d-4595-a328-a859dc31b021-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a56ff1b-d00d-4595-a328-a859dc31b021-plugins-config-volume\") pod \"7a56ff1b-d00d-4595-a328-a859dc31b021\" (UID: \"7a56ff1b-d00d-4595-a328-a859dc31b021\") "
Nov 01 07:03:34.824268 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:34.824246    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a56ff1b-d00d-4595-a328-a859dc31b021-plugins-config-volume" pod "7a56ff1b-d00d-4595-a328-a859dc31b021" (UID: "7a56ff1b-d00d-4595-a328-a859dc31b021"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:03:34.825900 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:34.825882    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a56ff1b-d00d-4595-a328-a859dc31b021-kube-api-access-z2ccc" pod "7a56ff1b-d00d-4595-a328-a859dc31b021" (UID: "7a56ff1b-d00d-4595-a328-a859dc31b021"). InnerVolumeSpecName "kube-api-access-z2ccc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:03:34.925609 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:34.925574    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2ccc\" (UniqueName: \"kubernetes.io/projected/7a56ff1b-d00d-4595-a328-a859dc31b021-kube-api-access-z2ccc\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:34.925609 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:34.925604    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a56ff1b-d00d-4595-a328-a859dc31b021-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:35.170970 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:35.170930    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7"]
Nov 01 07:03:35.174695 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:35.174671    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-main-045f69-fdfc9d8c-gkjc7"]
Nov 01 07:03:36.548294 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:36.548253    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:03:36.961234 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:36.961202    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a56ff1b-d00d-4595-a328-a859dc31b021" path="/var/lib/kubelet/pods/7a56ff1b-d00d-4595-a328-a859dc31b021/volumes"
Nov 01 07:03:38.592899 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:38.592855    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:03:42.307652 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:42.307614    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:03:43.089983 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.089941    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:03:43.134970 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.134937    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"]
Nov 01 07:03:43.135183 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.135158    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq" podUID="6fc66d3c-9dc5-4a05-908e-30240252019a" containerName="maas-api" containerID="cri-o://9ce930a06fc232cd6d07cc53caef039feae3b7f99bbe56cbae613c52f23cc116" gracePeriod=30
Nov 01 07:03:43.235075 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.235038    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:03:43.386855 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.386828    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"
Nov 01 07:03:43.410263 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.410235    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fc66d3c-9dc5-4a05-908e-30240252019a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fc66d3c-9dc5-4a05-908e-30240252019a-maas-api-tls\") pod \"6fc66d3c-9dc5-4a05-908e-30240252019a\" (UID: \"6fc66d3c-9dc5-4a05-908e-30240252019a\") "
Nov 01 07:03:43.410420 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.410278    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fc66d3c-9dc5-4a05-908e-30240252019a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fc66d3c-9dc5-4a05-908e-30240252019a-openshift-service-ca\") pod \"6fc66d3c-9dc5-4a05-908e-30240252019a\" (UID: \"6fc66d3c-9dc5-4a05-908e-30240252019a\") "
Nov 01 07:03:43.410420 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.410354    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc66d3c-9dc5-4a05-908e-30240252019a-kube-api-access-xwtx7\" (UniqueName: \"kubernetes.io/projected/6fc66d3c-9dc5-4a05-908e-30240252019a-kube-api-access-xwtx7\") pod \"6fc66d3c-9dc5-4a05-908e-30240252019a\" (UID: \"6fc66d3c-9dc5-4a05-908e-30240252019a\") "
Nov 01 07:03:43.410540 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.410488    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fc66d3c-9dc5-4a05-908e-30240252019a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fc66d3c-9dc5-4a05-908e-30240252019a-metrics-certs\") pod \"6fc66d3c-9dc5-4a05-908e-30240252019a\" (UID: \"6fc66d3c-9dc5-4a05-908e-30240252019a\") "
Nov 01 07:03:43.410699 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.410674    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc66d3c-9dc5-4a05-908e-30240252019a-openshift-service-ca" pod "6fc66d3c-9dc5-4a05-908e-30240252019a" (UID: "6fc66d3c-9dc5-4a05-908e-30240252019a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:03:43.410843 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.410827    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fc66d3c-9dc5-4a05-908e-30240252019a-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:43.412396 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.412375    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fc66d3c-9dc5-4a05-908e-30240252019a-maas-api-tls" pod "6fc66d3c-9dc5-4a05-908e-30240252019a" (UID: "6fc66d3c-9dc5-4a05-908e-30240252019a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:03:43.412396 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.412385    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc66d3c-9dc5-4a05-908e-30240252019a-kube-api-access-xwtx7" pod "6fc66d3c-9dc5-4a05-908e-30240252019a" (UID: "6fc66d3c-9dc5-4a05-908e-30240252019a"). InnerVolumeSpecName "kube-api-access-xwtx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:03:43.412503 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.412453    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fc66d3c-9dc5-4a05-908e-30240252019a-metrics-certs" pod "6fc66d3c-9dc5-4a05-908e-30240252019a" (UID: "6fc66d3c-9dc5-4a05-908e-30240252019a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:03:43.523181 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.523124    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fc66d3c-9dc5-4a05-908e-30240252019a-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:43.523442 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.523411    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwtx7\" (UniqueName: \"kubernetes.io/projected/6fc66d3c-9dc5-4a05-908e-30240252019a-kube-api-access-xwtx7\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:43.523595 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.523521    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fc66d3c-9dc5-4a05-908e-30240252019a-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:03:43.796855 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.796764    5619 generic.go:356] "Generic (PLEG): container finished" podID="6fc66d3c-9dc5-4a05-908e-30240252019a" containerID="9ce930a06fc232cd6d07cc53caef039feae3b7f99bbe56cbae613c52f23cc116" exitCode=0
Nov 01 07:03:43.796855 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.796821    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq" event={"ID":"6fc66d3c-9dc5-4a05-908e-30240252019a","Type":"ContainerDied","Data":"9ce930a06fc232cd6d07cc53caef039feae3b7f99bbe56cbae613c52f23cc116"}
Nov 01 07:03:43.797110 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.796874    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq" event={"ID":"6fc66d3c-9dc5-4a05-908e-30240252019a","Type":"ContainerDied","Data":"abcd45edb423e1983a246317701cba7e1721e80df89ebf5847e4267e1026e6f9"}
Nov 01 07:03:43.797110 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.796907    5619 scope.go:122] "RemoveContainer" containerID="9ce930a06fc232cd6d07cc53caef039feae3b7f99bbe56cbae613c52f23cc116"
Nov 01 07:03:43.797110 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.797056    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"
Nov 01 07:03:43.809993 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.809889    5619 scope.go:122] "RemoveContainer" containerID="9ce930a06fc232cd6d07cc53caef039feae3b7f99bbe56cbae613c52f23cc116"
Nov 01 07:03:43.810265 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:03:43.810231    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce930a06fc232cd6d07cc53caef039feae3b7f99bbe56cbae613c52f23cc116\": container with ID starting with 9ce930a06fc232cd6d07cc53caef039feae3b7f99bbe56cbae613c52f23cc116 not found: ID does not exist" containerID="9ce930a06fc232cd6d07cc53caef039feae3b7f99bbe56cbae613c52f23cc116"
Nov 01 07:03:43.810426 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.810276    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce930a06fc232cd6d07cc53caef039feae3b7f99bbe56cbae613c52f23cc116"} err="failed to get container status \"9ce930a06fc232cd6d07cc53caef039feae3b7f99bbe56cbae613c52f23cc116\": rpc error: code = NotFound desc = could not find container \"9ce930a06fc232cd6d07cc53caef039feae3b7f99bbe56cbae613c52f23cc116\": container with ID starting with 9ce930a06fc232cd6d07cc53caef039feae3b7f99bbe56cbae613c52f23cc116 not found: ID does not exist"
Nov 01 07:03:43.964117 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:43.964086    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"]
Nov 01 07:03:44.014764 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:44.014731    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-main-045f69-776cbd689c-ppmvq"]
Nov 01 07:03:44.960779 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:44.960749    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fc66d3c-9dc5-4a05-908e-30240252019a" path="/var/lib/kubelet/pods/6fc66d3c-9dc5-4a05-908e-30240252019a/volumes"
Nov 01 07:03:48.182142 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:48.182101    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:03:48.322213 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:48.322177    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:03:55.878500 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:03:55.878452    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:03:59.184509 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:59.184468    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:03:59.378185 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:59.378150    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:03:59.612793 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:03:59.612757    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:04:02.024867 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:02.024830    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:04:02.554544 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:02.554496    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:04:02.565349 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:02.565312    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"]
Nov 01 07:04:02.565552 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:02.565528    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b" podUID="e79bdaee-9cd1-4204-8f2f-d8d356851307" containerName="istio-proxy" containerID="cri-o://bf47b28478e293ff0f54efa624cd81d8207df5f503d0c711b7853cd283243114" gracePeriod=30
Nov 01 07:04:05.222440 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:05.222402    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:04:05.412059 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:05.411995    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:04:07.822036 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.822013    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 07:04:07.882594 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.882568    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-credential-socket\") pod \"e79bdaee-9cd1-4204-8f2f-d8d356851307\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") "
Nov 01 07:04:07.882724 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.882632    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-token\" (UniqueName: \"kubernetes.io/projected/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-token\") pod \"e79bdaee-9cd1-4204-8f2f-d8d356851307\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") "
Nov 01 07:04:07.882724 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.882701    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e79bdaee-9cd1-4204-8f2f-d8d356851307-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e79bdaee-9cd1-4204-8f2f-d8d356851307-istiod-ca-cert\") pod \"e79bdaee-9cd1-4204-8f2f-d8d356851307\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") "
Nov 01 07:04:07.882816 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.882724    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e79bdaee-9cd1-4204-8f2f-d8d356851307-kube-api-access-bh72n\" (UniqueName: \"kubernetes.io/projected/e79bdaee-9cd1-4204-8f2f-d8d356851307-kube-api-access-bh72n\") pod \"e79bdaee-9cd1-4204-8f2f-d8d356851307\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") "
Nov 01 07:04:07.882816 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.882772    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-workload-certs\") pod \"e79bdaee-9cd1-4204-8f2f-d8d356851307\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") "
Nov 01 07:04:07.882904 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.882808    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-credential-socket" pod "e79bdaee-9cd1-4204-8f2f-d8d356851307" (UID: "e79bdaee-9cd1-4204-8f2f-d8d356851307"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:04:07.882904 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.882834    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-podinfo\") pod \"e79bdaee-9cd1-4204-8f2f-d8d356851307\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") "
Nov 01 07:04:07.882904 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.882874    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-workload-socket\") pod \"e79bdaee-9cd1-4204-8f2f-d8d356851307\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") "
Nov 01 07:04:07.883047 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.882906    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-envoy\") pod \"e79bdaee-9cd1-4204-8f2f-d8d356851307\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") "
Nov 01 07:04:07.883047 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.882947    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-data\") pod \"e79bdaee-9cd1-4204-8f2f-d8d356851307\" (UID: \"e79bdaee-9cd1-4204-8f2f-d8d356851307\") "
Nov 01 07:04:07.883157 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.883067    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-workload-certs" pod "e79bdaee-9cd1-4204-8f2f-d8d356851307" (UID: "e79bdaee-9cd1-4204-8f2f-d8d356851307"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:04:07.883157 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.883076    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e79bdaee-9cd1-4204-8f2f-d8d356851307-istiod-ca-cert" pod "e79bdaee-9cd1-4204-8f2f-d8d356851307" (UID: "e79bdaee-9cd1-4204-8f2f-d8d356851307"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:04:07.883427 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.883348    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-data" pod "e79bdaee-9cd1-4204-8f2f-d8d356851307" (UID: "e79bdaee-9cd1-4204-8f2f-d8d356851307"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:04:07.883427 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.883359    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-workload-socket" pod "e79bdaee-9cd1-4204-8f2f-d8d356851307" (UID: "e79bdaee-9cd1-4204-8f2f-d8d356851307"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:04:07.883596 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.883488    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e79bdaee-9cd1-4204-8f2f-d8d356851307-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:07.883596 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.883511    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:07.883596 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.883528    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:07.883596 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.883543    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:07.883596 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.883558    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:07.884950 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.884922    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-token" pod "e79bdaee-9cd1-4204-8f2f-d8d356851307" (UID: "e79bdaee-9cd1-4204-8f2f-d8d356851307"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:04:07.885057 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.884983    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79bdaee-9cd1-4204-8f2f-d8d356851307-kube-api-access-bh72n" pod "e79bdaee-9cd1-4204-8f2f-d8d356851307" (UID: "e79bdaee-9cd1-4204-8f2f-d8d356851307"). InnerVolumeSpecName "kube-api-access-bh72n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:04:07.885057 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.885024    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-podinfo" pod "e79bdaee-9cd1-4204-8f2f-d8d356851307" (UID: "e79bdaee-9cd1-4204-8f2f-d8d356851307"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:04:07.885139 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.885056    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-envoy" pod "e79bdaee-9cd1-4204-8f2f-d8d356851307" (UID: "e79bdaee-9cd1-4204-8f2f-d8d356851307"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:04:07.913745 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.913721    5619 generic.go:356] "Generic (PLEG): container finished" podID="e79bdaee-9cd1-4204-8f2f-d8d356851307" containerID="bf47b28478e293ff0f54efa624cd81d8207df5f503d0c711b7853cd283243114" exitCode=0
Nov 01 07:04:07.913825 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.913766    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b" event={"ID":"e79bdaee-9cd1-4204-8f2f-d8d356851307","Type":"ContainerDied","Data":"bf47b28478e293ff0f54efa624cd81d8207df5f503d0c711b7853cd283243114"}
Nov 01 07:04:07.913825 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.913804    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b" event={"ID":"e79bdaee-9cd1-4204-8f2f-d8d356851307","Type":"ContainerDied","Data":"92bc40bb5e75a7bb03314292d2f1cf4267444d6efa97d64def1d31bf30c946d0"}
Nov 01 07:04:07.913910 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.913826    5619 scope.go:122] "RemoveContainer" containerID="bf47b28478e293ff0f54efa624cd81d8207df5f503d0c711b7853cd283243114"
Nov 01 07:04:07.913910 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.913769    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"
Nov 01 07:04:07.923124 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.923107    5619 scope.go:122] "RemoveContainer" containerID="6294debcf630b144d3526c3b4ff89e02ed7c2d16ed2c80c34ffd401b881ec457"
Nov 01 07:04:07.930766 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.930750    5619 scope.go:122] "RemoveContainer" containerID="bf47b28478e293ff0f54efa624cd81d8207df5f503d0c711b7853cd283243114"
Nov 01 07:04:07.931003 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:04:07.930984    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf47b28478e293ff0f54efa624cd81d8207df5f503d0c711b7853cd283243114\": container with ID starting with bf47b28478e293ff0f54efa624cd81d8207df5f503d0c711b7853cd283243114 not found: ID does not exist" containerID="bf47b28478e293ff0f54efa624cd81d8207df5f503d0c711b7853cd283243114"
Nov 01 07:04:07.931057 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.931011    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf47b28478e293ff0f54efa624cd81d8207df5f503d0c711b7853cd283243114"} err="failed to get container status \"bf47b28478e293ff0f54efa624cd81d8207df5f503d0c711b7853cd283243114\": rpc error: code = NotFound desc = could not find container \"bf47b28478e293ff0f54efa624cd81d8207df5f503d0c711b7853cd283243114\": container with ID starting with bf47b28478e293ff0f54efa624cd81d8207df5f503d0c711b7853cd283243114 not found: ID does not exist"
Nov 01 07:04:07.931057 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.931027    5619 scope.go:122] "RemoveContainer" containerID="6294debcf630b144d3526c3b4ff89e02ed7c2d16ed2c80c34ffd401b881ec457"
Nov 01 07:04:07.931241 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:04:07.931221    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6294debcf630b144d3526c3b4ff89e02ed7c2d16ed2c80c34ffd401b881ec457\": container with ID starting with 6294debcf630b144d3526c3b4ff89e02ed7c2d16ed2c80c34ffd401b881ec457 not found: ID does not exist" containerID="6294debcf630b144d3526c3b4ff89e02ed7c2d16ed2c80c34ffd401b881ec457"
Nov 01 07:04:07.931282 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.931248    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6294debcf630b144d3526c3b4ff89e02ed7c2d16ed2c80c34ffd401b881ec457"} err="failed to get container status \"6294debcf630b144d3526c3b4ff89e02ed7c2d16ed2c80c34ffd401b881ec457\": rpc error: code = NotFound desc = could not find container \"6294debcf630b144d3526c3b4ff89e02ed7c2d16ed2c80c34ffd401b881ec457\": container with ID starting with 6294debcf630b144d3526c3b4ff89e02ed7c2d16ed2c80c34ffd401b881ec457 not found: ID does not exist"
Nov 01 07:04:07.985266 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.985206    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh72n\" (UniqueName: \"kubernetes.io/projected/e79bdaee-9cd1-4204-8f2f-d8d356851307-kube-api-access-bh72n\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:07.985266 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.985234    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:07.985480 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.985274    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:07.985480 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:07.985289    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e79bdaee-9cd1-4204-8f2f-d8d356851307-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:04:08.083338 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:08.083293    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"]
Nov 01 07:04:08.087618 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:08.087595    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-main-045f69-openshift-default-7df876676f-qtd5b"]
Nov 01 07:04:08.960609 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:08.960579    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e79bdaee-9cd1-4204-8f2f-d8d356851307" path="/var/lib/kubelet/pods/e79bdaee-9cd1-4204-8f2f-d8d356851307/volumes"
Nov 01 07:04:25.878776 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:04:25.878729    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:04:26.982858 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:26.982817    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:04:27.472943 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.472909    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"]
Nov 01 07:04:27.473419 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.473399    5619 cpu_manager.go:414] "containerMap: removing container" podUID="e79bdaee-9cd1-4204-8f2f-d8d356851307" containerName="istio-proxy"
Nov 01 07:04:27.473542 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.473422    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79bdaee-9cd1-4204-8f2f-d8d356851307" containerName="istio-proxy"
Nov 01 07:04:27.473542 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.473438    5619 cpu_manager.go:414] "containerMap: removing container" podUID="e79bdaee-9cd1-4204-8f2f-d8d356851307" containerName="istio-proxy"
Nov 01 07:04:27.473542 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.473444    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79bdaee-9cd1-4204-8f2f-d8d356851307" containerName="istio-proxy"
Nov 01 07:04:27.473542 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.473454    5619 cpu_manager.go:414] "containerMap: removing container" podUID="7a56ff1b-d00d-4595-a328-a859dc31b021" containerName="payload-processing"
Nov 01 07:04:27.473542 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.473461    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a56ff1b-d00d-4595-a328-a859dc31b021" containerName="payload-processing"
Nov 01 07:04:27.473542 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.473484    5619 cpu_manager.go:414] "containerMap: removing container" podUID="6fc66d3c-9dc5-4a05-908e-30240252019a" containerName="maas-api"
Nov 01 07:04:27.473542 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.473489    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc66d3c-9dc5-4a05-908e-30240252019a" containerName="maas-api"
Nov 01 07:04:27.473774 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.473569    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="e79bdaee-9cd1-4204-8f2f-d8d356851307" containerName="istio-proxy"
Nov 01 07:04:27.473774 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.473577    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="e79bdaee-9cd1-4204-8f2f-d8d356851307" containerName="istio-proxy"
Nov 01 07:04:27.473774 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.473584    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a56ff1b-d00d-4595-a328-a859dc31b021" containerName="payload-processing"
Nov 01 07:04:27.473774 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.473591    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc66d3c-9dc5-4a05-908e-30240252019a" containerName="maas-api"
Nov 01 07:04:27.476668 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.476653    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.530350 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.530304    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9af193c3-openshift-default-dockercfg-npf5g\""
Nov 01 07:04:27.567104 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.567073    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-envoy\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.567263 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.567108    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-workload-certs\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.567263 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.567199    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbeeaa26-7a36-4313-b609-17cdec1b0632-istiod-ca-cert\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.567263 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.567240    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-data\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.567411 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.567372    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-workload-socket\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.567453 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.567411    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-podinfo\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.567491 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.567473    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rfzb\" (UniqueName: \"kubernetes.io/projected/dbeeaa26-7a36-4313-b609-17cdec1b0632-kube-api-access-8rfzb\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.567524 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.567497    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-token\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.567560 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.567535    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-credential-socket\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.572500 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.572474    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:04:27.583451 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.583419    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"]
Nov 01 07:04:27.668933 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.668900    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-token\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.669122 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.668941    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-credential-socket\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.669122 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.669010    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-envoy\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.669122 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.669038    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-workload-certs\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.669122 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.669077    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbeeaa26-7a36-4313-b609-17cdec1b0632-istiod-ca-cert\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.669122 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.669103    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-data\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.669445 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.669162    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-workload-socket\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.669445 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.669247    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-podinfo\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.669445 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.669283    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rfzb\" (UniqueName: \"kubernetes.io/projected/dbeeaa26-7a36-4313-b609-17cdec1b0632-kube-api-access-8rfzb\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.669618 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.669576    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-credential-socket\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.669682 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.669657    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-data\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.669745 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.669717    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-workload-socket\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.669858 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.669839    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-workload-certs\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.669934 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.669917    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbeeaa26-7a36-4313-b609-17cdec1b0632-istiod-ca-cert\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.671425 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.671406    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-envoy\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.671675 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.671657    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-podinfo\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.730894 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.730832    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-token\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.731013 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.730906    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rfzb\" (UniqueName: \"kubernetes.io/projected/dbeeaa26-7a36-4313-b609-17cdec1b0632-kube-api-access-8rfzb\") pod \"e2e-ait-9af193c3-openshift-default-688585f499-7nbl5\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") " pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.790561 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.790296    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:27.848589 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:27.848560    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:04:28.222790 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:28.222767    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"]
Nov 01 07:04:28.224840 ip-10-0-129-213 kubenswrapper[5619]: W1101 07:04:28.224810    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbeeaa26_7a36_4313_b609_17cdec1b0632.slice/crio-f67a20001383f3af5f97ab6e1b585e7021f157413110dcfd7890a603ebb7811c WatchSource:0}: Error finding container f67a20001383f3af5f97ab6e1b585e7021f157413110dcfd7890a603ebb7811c: Status 404 returned error can't find the container with id f67a20001383f3af5f97ab6e1b585e7021f157413110dcfd7890a603ebb7811c
Nov 01 07:04:28.226931 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:28.226903    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 07:04:28.227007 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:28.226967    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 07:04:28.227007 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:28.226995    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 07:04:29.015476 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:29.015442    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5" event={"ID":"dbeeaa26-7a36-4313-b609-17cdec1b0632","Type":"ContainerStarted","Data":"b489ff8728f8e28db769cab4f535d011aa49f7e193afb21c07db0afb6bfb7994"}
Nov 01 07:04:29.015643 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:29.015490    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5" event={"ID":"dbeeaa26-7a36-4313-b609-17cdec1b0632","Type":"ContainerStarted","Data":"f67a20001383f3af5f97ab6e1b585e7021f157413110dcfd7890a603ebb7811c"}
Nov 01 07:04:29.190675 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:29.190625    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5" podStartSLOduration=2.190611985 podStartE2EDuration="2.190611985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:04:29.18787368 +0000 UTC m=+4706.398697223" watchObservedRunningTime="2026-11-01 07:04:29.190611985 +0000 UTC m=+4706.401435494"
Nov 01 07:04:29.791035 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:29.790995    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:29.796144 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:29.796120    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:30.020213 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:30.020175    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:30.021040 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:30.021018    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:31.188042 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:31.188003    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:04:31.397460 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:31.397426    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:04:34.426911 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:34.426876    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:04:34.600069 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:34.600037    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:04:34.993150 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:34.993120    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb"]
Nov 01 07:04:34.996815 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:34.996797    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb"
Nov 01 07:04:35.094158 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:35.094131    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9af193c3-dockercfg-r4q58\""
Nov 01 07:04:35.094313 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:35.094204    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9af193c3\""
Nov 01 07:04:35.096658 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:35.096638    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb"]
Nov 01 07:04:35.148734 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:35.148706    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhsbn\" (UniqueName: \"kubernetes.io/projected/2baa4c29-73e2-4d84-8d09-880513f23eb8-kube-api-access-zhsbn\") pod \"payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb\" (UID: \"2baa4c29-73e2-4d84-8d09-880513f23eb8\") " pod="openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb"
Nov 01 07:04:35.148871 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:35.148831    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2baa4c29-73e2-4d84-8d09-880513f23eb8-plugins-config-volume\") pod \"payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb\" (UID: \"2baa4c29-73e2-4d84-8d09-880513f23eb8\") " pod="openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb"
Nov 01 07:04:35.249507 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:35.249439    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhsbn\" (UniqueName: \"kubernetes.io/projected/2baa4c29-73e2-4d84-8d09-880513f23eb8-kube-api-access-zhsbn\") pod \"payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb\" (UID: \"2baa4c29-73e2-4d84-8d09-880513f23eb8\") " pod="openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb"
Nov 01 07:04:35.249659 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:35.249528    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2baa4c29-73e2-4d84-8d09-880513f23eb8-plugins-config-volume\") pod \"payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb\" (UID: \"2baa4c29-73e2-4d84-8d09-880513f23eb8\") " pod="openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb"
Nov 01 07:04:35.250163 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:35.250145    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2baa4c29-73e2-4d84-8d09-880513f23eb8-plugins-config-volume\") pod \"payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb\" (UID: \"2baa4c29-73e2-4d84-8d09-880513f23eb8\") " pod="openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb"
Nov 01 07:04:35.311379 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:35.311352    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhsbn\" (UniqueName: \"kubernetes.io/projected/2baa4c29-73e2-4d84-8d09-880513f23eb8-kube-api-access-zhsbn\") pod \"payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb\" (UID: \"2baa4c29-73e2-4d84-8d09-880513f23eb8\") " pod="openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb"
Nov 01 07:04:35.610710 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:35.610672    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb"
Nov 01 07:04:35.984658 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:35.984634    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb"]
Nov 01 07:04:35.986348 ip-10-0-129-213 kubenswrapper[5619]: W1101 07:04:35.986305    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2baa4c29_73e2_4d84_8d09_880513f23eb8.slice/crio-f0a84153e137e20e90b3208b3a0a5fd95a89e37ca5f113bb7388dc7416a62831 WatchSource:0}: Error finding container f0a84153e137e20e90b3208b3a0a5fd95a89e37ca5f113bb7388dc7416a62831: Status 404 returned error can't find the container with id f0a84153e137e20e90b3208b3a0a5fd95a89e37ca5f113bb7388dc7416a62831
Nov 01 07:04:36.048892 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:36.048864    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb" event={"ID":"2baa4c29-73e2-4d84-8d09-880513f23eb8","Type":"ContainerStarted","Data":"3b81501cb61e31d283eeafde4bb6db5ecb120e475eed6ddbf8e8fef8b7d1f1f0"}
Nov 01 07:04:36.049006 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:36.048917    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb" event={"ID":"2baa4c29-73e2-4d84-8d09-880513f23eb8","Type":"ContainerStarted","Data":"f0a84153e137e20e90b3208b3a0a5fd95a89e37ca5f113bb7388dc7416a62831"}
Nov 01 07:04:36.049006 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:36.048960    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb"
Nov 01 07:04:36.167354 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:36.167294    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb" podStartSLOduration=2.167280758 podStartE2EDuration="2.167280758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:04:36.165626421 +0000 UTC m=+4713.376449929" watchObservedRunningTime="2026-11-01 07:04:36.167280758 +0000 UTC m=+4713.378104264"
Nov 01 07:04:37.509767 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:37.509732    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:04:37.703582 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:37.703542    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:04:38.945361 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:38.945316    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:04:42.373601 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:42.373553    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:04:43.441383 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:43.441339    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:04:47.066557 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:47.066523    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb"
Nov 01 07:04:47.208971 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:47.208932    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:04:51.589726 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:51.589675    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:04:52.011240 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:52.011206    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:04:55.149134 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:55.149046    5619 generic.go:356] "Generic (PLEG): container finished" podID="dbeeaa26-7a36-4313-b609-17cdec1b0632" containerID="b489ff8728f8e28db769cab4f535d011aa49f7e193afb21c07db0afb6bfb7994" exitCode=0
Nov 01 07:04:55.149573 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:55.149125    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5" event={"ID":"dbeeaa26-7a36-4313-b609-17cdec1b0632","Type":"ContainerDied","Data":"b489ff8728f8e28db769cab4f535d011aa49f7e193afb21c07db0afb6bfb7994"}
Nov 01 07:04:55.149573 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:55.149509    5619 scope.go:122] "RemoveContainer" containerID="b489ff8728f8e28db769cab4f535d011aa49f7e193afb21c07db0afb6bfb7994"
Nov 01 07:04:55.150679 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:55.150653    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 07:04:55.150755 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:55.150724    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 07:04:55.150796 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:55.150760    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 07:04:55.897865 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:04:55.897828    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:04:56.179075 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:56.178985    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5" event={"ID":"dbeeaa26-7a36-4313-b609-17cdec1b0632","Type":"ContainerStarted","Data":"b56f9987d3f528405ceca05407ddf7086e65d92fd65608b44c73ee35fc84f024"}
Nov 01 07:04:56.790969 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:56.790932    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:04:56.792308 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:56.792272    5619 patch_prober.go:29] interesting pod/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 01 07:04:56.792482 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:56.792362    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5" podUID="dbeeaa26-7a36-4313-b609-17cdec1b0632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 01 07:04:57.106212 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:57.106175    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:04:57.791544 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:57.791511    5619 patch_prober.go:29] interesting pod/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 01 07:04:57.791944 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:57.791581    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5" podUID="dbeeaa26-7a36-4313-b609-17cdec1b0632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 01 07:04:58.790862 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:58.790825    5619 patch_prober.go:29] interesting pod/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 01 07:04:58.791047 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:58.790885    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5" podUID="dbeeaa26-7a36-4313-b609-17cdec1b0632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 01 07:04:59.791550 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:59.791515    5619 patch_prober.go:29] interesting pod/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 01 07:04:59.792029 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:04:59.791582    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5" podUID="dbeeaa26-7a36-4313-b609-17cdec1b0632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 01 07:05:00.020884 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:00.020850    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:05:00.791695 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:00.791655    5619 patch_prober.go:29] interesting pod/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 01 07:05:00.792184 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:00.791715    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5" podUID="dbeeaa26-7a36-4313-b609-17cdec1b0632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 01 07:05:01.795625 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:01.795589    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:05:01.796625 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:01.796601    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:05:01.868767 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:01.868735    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:05:06.660099 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:06.660059    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:05:11.729789 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:11.729758    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:05:16.558397 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:16.558361    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:05:21.263080 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:21.263035    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:05:25.878930 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:05:25.878897    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:05:26.069625 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:26.069587    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:05:29.194282 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:29.194246    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb"]
Nov 01 07:05:29.555382 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:29.555296    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:05:29.694862 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:29.694823    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:05:30.759917 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:30.759870    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:05:34.195457 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:34.195389    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb" podUID="2baa4c29-73e2-4d84-8d09-880513f23eb8" containerName="payload-processing" containerID="cri-o://3b81501cb61e31d283eeafde4bb6db5ecb120e475eed6ddbf8e8fef8b7d1f1f0" gracePeriod=25
Nov 01 07:05:34.366011 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:34.365963    5619 generic.go:356] "Generic (PLEG): container finished" podID="2baa4c29-73e2-4d84-8d09-880513f23eb8" containerID="3b81501cb61e31d283eeafde4bb6db5ecb120e475eed6ddbf8e8fef8b7d1f1f0" exitCode=0
Nov 01 07:05:34.366177 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:34.366118    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb" event={"ID":"2baa4c29-73e2-4d84-8d09-880513f23eb8","Type":"ContainerDied","Data":"3b81501cb61e31d283eeafde4bb6db5ecb120e475eed6ddbf8e8fef8b7d1f1f0"}
Nov 01 07:05:34.443979 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:34.443955    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb"
Nov 01 07:05:34.464083 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:34.464018    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2baa4c29-73e2-4d84-8d09-880513f23eb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2baa4c29-73e2-4d84-8d09-880513f23eb8-plugins-config-volume\") pod \"2baa4c29-73e2-4d84-8d09-880513f23eb8\" (UID: \"2baa4c29-73e2-4d84-8d09-880513f23eb8\") "
Nov 01 07:05:34.464229 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:34.464213    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2baa4c29-73e2-4d84-8d09-880513f23eb8-kube-api-access-zhsbn\" (UniqueName: \"kubernetes.io/projected/2baa4c29-73e2-4d84-8d09-880513f23eb8-kube-api-access-zhsbn\") pod \"2baa4c29-73e2-4d84-8d09-880513f23eb8\" (UID: \"2baa4c29-73e2-4d84-8d09-880513f23eb8\") "
Nov 01 07:05:34.464405 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:34.464380    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2baa4c29-73e2-4d84-8d09-880513f23eb8-plugins-config-volume" pod "2baa4c29-73e2-4d84-8d09-880513f23eb8" (UID: "2baa4c29-73e2-4d84-8d09-880513f23eb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:05:34.464508 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:34.464485    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2baa4c29-73e2-4d84-8d09-880513f23eb8-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:05:34.466167 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:34.466142    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2baa4c29-73e2-4d84-8d09-880513f23eb8-kube-api-access-zhsbn" pod "2baa4c29-73e2-4d84-8d09-880513f23eb8" (UID: "2baa4c29-73e2-4d84-8d09-880513f23eb8"). InnerVolumeSpecName "kube-api-access-zhsbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:05:34.565372 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:34.565310    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhsbn\" (UniqueName: \"kubernetes.io/projected/2baa4c29-73e2-4d84-8d09-880513f23eb8-kube-api-access-zhsbn\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:05:34.919764 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:34.919728    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:05:35.371960 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:35.371928    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb"
Nov 01 07:05:35.372419 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:35.371931    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb" event={"ID":"2baa4c29-73e2-4d84-8d09-880513f23eb8","Type":"ContainerDied","Data":"f0a84153e137e20e90b3208b3a0a5fd95a89e37ca5f113bb7388dc7416a62831"}
Nov 01 07:05:35.372419 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:35.372083    5619 scope.go:122] "RemoveContainer" containerID="3b81501cb61e31d283eeafde4bb6db5ecb120e475eed6ddbf8e8fef8b7d1f1f0"
Nov 01 07:05:35.539092 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:35.539057    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb"]
Nov 01 07:05:35.589966 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:35.589934    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9af193c3-785fbc4d8f-gmjnb"]
Nov 01 07:05:36.400345 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:36.400294    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:05:36.678040 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:36.677957    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:05:36.961157 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:36.961080    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2baa4c29-73e2-4d84-8d09-880513f23eb8" path="/var/lib/kubelet/pods/2baa4c29-73e2-4d84-8d09-880513f23eb8/volumes"
Nov 01 07:05:38.849399 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:38.849355    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:05:38.851248 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:38.851221    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"]
Nov 01 07:05:38.851480 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:38.851455    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5" podUID="dbeeaa26-7a36-4313-b609-17cdec1b0632" containerName="istio-proxy" containerID="cri-o://b56f9987d3f528405ceca05407ddf7086e65d92fd65608b44c73ee35fc84f024" gracePeriod=30
Nov 01 07:05:40.715447 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:40.715405    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:05:43.783931 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:43.783888    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:05:44.100912 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.100884    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:05:44.164103 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.164070    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-workload-socket\") pod \"dbeeaa26-7a36-4313-b609-17cdec1b0632\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") "
Nov 01 07:05:44.164278 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.164125    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-envoy\") pod \"dbeeaa26-7a36-4313-b609-17cdec1b0632\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") "
Nov 01 07:05:44.164278 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.164152    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-data\") pod \"dbeeaa26-7a36-4313-b609-17cdec1b0632\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") "
Nov 01 07:05:44.164278 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.164185    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbeeaa26-7a36-4313-b609-17cdec1b0632-kube-api-access-8rfzb\" (UniqueName: \"kubernetes.io/projected/dbeeaa26-7a36-4313-b609-17cdec1b0632-kube-api-access-8rfzb\") pod \"dbeeaa26-7a36-4313-b609-17cdec1b0632\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") "
Nov 01 07:05:44.164278 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.164258    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-podinfo\") pod \"dbeeaa26-7a36-4313-b609-17cdec1b0632\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") "
Nov 01 07:05:44.164566 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.164290    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-workload-certs\") pod \"dbeeaa26-7a36-4313-b609-17cdec1b0632\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") "
Nov 01 07:05:44.164566 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.164353    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-workload-socket" pod "dbeeaa26-7a36-4313-b609-17cdec1b0632" (UID: "dbeeaa26-7a36-4313-b609-17cdec1b0632"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:05:44.164566 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.164404    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-token\" (UniqueName: \"kubernetes.io/projected/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-token\") pod \"dbeeaa26-7a36-4313-b609-17cdec1b0632\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") "
Nov 01 07:05:44.164566 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.164497    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbeeaa26-7a36-4313-b609-17cdec1b0632-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbeeaa26-7a36-4313-b609-17cdec1b0632-istiod-ca-cert\") pod \"dbeeaa26-7a36-4313-b609-17cdec1b0632\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") "
Nov 01 07:05:44.164566 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.164515    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-data" pod "dbeeaa26-7a36-4313-b609-17cdec1b0632" (UID: "dbeeaa26-7a36-4313-b609-17cdec1b0632"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:05:44.164566 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.164546    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-credential-socket\") pod \"dbeeaa26-7a36-4313-b609-17cdec1b0632\" (UID: \"dbeeaa26-7a36-4313-b609-17cdec1b0632\") "
Nov 01 07:05:44.164913 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.164608    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-workload-certs" pod "dbeeaa26-7a36-4313-b609-17cdec1b0632" (UID: "dbeeaa26-7a36-4313-b609-17cdec1b0632"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:05:44.164913 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.164818    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbeeaa26-7a36-4313-b609-17cdec1b0632-istiod-ca-cert" pod "dbeeaa26-7a36-4313-b609-17cdec1b0632" (UID: "dbeeaa26-7a36-4313-b609-17cdec1b0632"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:05:44.164913 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.164887    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-credential-socket" pod "dbeeaa26-7a36-4313-b609-17cdec1b0632" (UID: "dbeeaa26-7a36-4313-b609-17cdec1b0632"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:05:44.165167 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.164962    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbeeaa26-7a36-4313-b609-17cdec1b0632-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:05:44.165167 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.164977    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:05:44.165167 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.164986    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:05:44.165167 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.164997    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:05:44.166704 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.166676    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-token" pod "dbeeaa26-7a36-4313-b609-17cdec1b0632" (UID: "dbeeaa26-7a36-4313-b609-17cdec1b0632"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:05:44.166704 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.166684    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-envoy" pod "dbeeaa26-7a36-4313-b609-17cdec1b0632" (UID: "dbeeaa26-7a36-4313-b609-17cdec1b0632"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:05:44.166877 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.166844    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbeeaa26-7a36-4313-b609-17cdec1b0632-kube-api-access-8rfzb" pod "dbeeaa26-7a36-4313-b609-17cdec1b0632" (UID: "dbeeaa26-7a36-4313-b609-17cdec1b0632"). InnerVolumeSpecName "kube-api-access-8rfzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:05:44.166970 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.166955    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-podinfo" pod "dbeeaa26-7a36-4313-b609-17cdec1b0632" (UID: "dbeeaa26-7a36-4313-b609-17cdec1b0632"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:05:44.266462 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.266427    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:05:44.266462 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.266459    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:05:44.266462 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.266469    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:05:44.266669 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.266479    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rfzb\" (UniqueName: \"kubernetes.io/projected/dbeeaa26-7a36-4313-b609-17cdec1b0632-kube-api-access-8rfzb\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:05:44.266669 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.266489    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbeeaa26-7a36-4313-b609-17cdec1b0632-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:05:44.416036 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.415996    5619 generic.go:356] "Generic (PLEG): container finished" podID="dbeeaa26-7a36-4313-b609-17cdec1b0632" containerID="b56f9987d3f528405ceca05407ddf7086e65d92fd65608b44c73ee35fc84f024" exitCode=0
Nov 01 07:05:44.416235 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.416064    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"
Nov 01 07:05:44.416235 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.416092    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5" event={"ID":"dbeeaa26-7a36-4313-b609-17cdec1b0632","Type":"ContainerDied","Data":"b56f9987d3f528405ceca05407ddf7086e65d92fd65608b44c73ee35fc84f024"}
Nov 01 07:05:44.416235 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.416150    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5" event={"ID":"dbeeaa26-7a36-4313-b609-17cdec1b0632","Type":"ContainerDied","Data":"f67a20001383f3af5f97ab6e1b585e7021f157413110dcfd7890a603ebb7811c"}
Nov 01 07:05:44.416235 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.416173    5619 scope.go:122] "RemoveContainer" containerID="b56f9987d3f528405ceca05407ddf7086e65d92fd65608b44c73ee35fc84f024"
Nov 01 07:05:44.427373 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.427315    5619 scope.go:122] "RemoveContainer" containerID="b489ff8728f8e28db769cab4f535d011aa49f7e193afb21c07db0afb6bfb7994"
Nov 01 07:05:44.435370 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.435350    5619 scope.go:122] "RemoveContainer" containerID="b56f9987d3f528405ceca05407ddf7086e65d92fd65608b44c73ee35fc84f024"
Nov 01 07:05:44.435628 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:05:44.435608    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b56f9987d3f528405ceca05407ddf7086e65d92fd65608b44c73ee35fc84f024\": container with ID starting with b56f9987d3f528405ceca05407ddf7086e65d92fd65608b44c73ee35fc84f024 not found: ID does not exist" containerID="b56f9987d3f528405ceca05407ddf7086e65d92fd65608b44c73ee35fc84f024"
Nov 01 07:05:44.435679 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.435638    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56f9987d3f528405ceca05407ddf7086e65d92fd65608b44c73ee35fc84f024"} err="failed to get container status \"b56f9987d3f528405ceca05407ddf7086e65d92fd65608b44c73ee35fc84f024\": rpc error: code = NotFound desc = could not find container \"b56f9987d3f528405ceca05407ddf7086e65d92fd65608b44c73ee35fc84f024\": container with ID starting with b56f9987d3f528405ceca05407ddf7086e65d92fd65608b44c73ee35fc84f024 not found: ID does not exist"
Nov 01 07:05:44.435679 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.435655    5619 scope.go:122] "RemoveContainer" containerID="b489ff8728f8e28db769cab4f535d011aa49f7e193afb21c07db0afb6bfb7994"
Nov 01 07:05:44.435913 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:05:44.435895    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b489ff8728f8e28db769cab4f535d011aa49f7e193afb21c07db0afb6bfb7994\": container with ID starting with b489ff8728f8e28db769cab4f535d011aa49f7e193afb21c07db0afb6bfb7994 not found: ID does not exist" containerID="b489ff8728f8e28db769cab4f535d011aa49f7e193afb21c07db0afb6bfb7994"
Nov 01 07:05:44.435956 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.435918    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b489ff8728f8e28db769cab4f535d011aa49f7e193afb21c07db0afb6bfb7994"} err="failed to get container status \"b489ff8728f8e28db769cab4f535d011aa49f7e193afb21c07db0afb6bfb7994\": rpc error: code = NotFound desc = could not find container \"b489ff8728f8e28db769cab4f535d011aa49f7e193afb21c07db0afb6bfb7994\": container with ID starting with b489ff8728f8e28db769cab4f535d011aa49f7e193afb21c07db0afb6bfb7994 not found: ID does not exist"
Nov 01 07:05:44.594743 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.594710    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"]
Nov 01 07:05:44.597059 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.597029    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9af193c3-openshift-default-688585f499-7nbl5"]
Nov 01 07:05:44.960515 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:05:44.960487    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbeeaa26-7a36-4313-b609-17cdec1b0632" path="/var/lib/kubelet/pods/dbeeaa26-7a36-4313-b609-17cdec1b0632/volumes"
Nov 01 07:05:55.878523 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:05:55.878478    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:06:25.880291 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:06:25.880247    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:06:39.700946 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:06:39.700908    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:06:44.138893 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:06:44.138851    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:06:48.476887 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:06:48.476846    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:06:55.878841 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:06:55.878800    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:07:20.754299 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.754259    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"]
Nov 01 07:07:20.754896 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.754874    5619 cpu_manager.go:414] "containerMap: removing container" podUID="dbeeaa26-7a36-4313-b609-17cdec1b0632" containerName="istio-proxy"
Nov 01 07:07:20.754942 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.754899    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbeeaa26-7a36-4313-b609-17cdec1b0632" containerName="istio-proxy"
Nov 01 07:07:20.754942 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.754930    5619 cpu_manager.go:414] "containerMap: removing container" podUID="2baa4c29-73e2-4d84-8d09-880513f23eb8" containerName="payload-processing"
Nov 01 07:07:20.754942 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.754938    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2baa4c29-73e2-4d84-8d09-880513f23eb8" containerName="payload-processing"
Nov 01 07:07:20.755046 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.754970    5619 cpu_manager.go:414] "containerMap: removing container" podUID="dbeeaa26-7a36-4313-b609-17cdec1b0632" containerName="istio-proxy"
Nov 01 07:07:20.755046 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.754979    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbeeaa26-7a36-4313-b609-17cdec1b0632" containerName="istio-proxy"
Nov 01 07:07:20.755113 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.755069    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbeeaa26-7a36-4313-b609-17cdec1b0632" containerName="istio-proxy"
Nov 01 07:07:20.755113 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.755084    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="2baa4c29-73e2-4d84-8d09-880513f23eb8" containerName="payload-processing"
Nov 01 07:07:20.755113 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.755098    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbeeaa26-7a36-4313-b609-17cdec1b0632" containerName="istio-proxy"
Nov 01 07:07:20.758182 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.758165    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:20.836458 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.836420    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-tmp-dir\") pod \"e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") " pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:20.836625 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.836468    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-model-cache\") pod \"e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") " pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:20.836625 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.836573    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-home\") pod \"e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") " pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:20.836716 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.836659    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d26e486-242a-4a70-9c30-1481063fb109-tls-certs\") pod \"e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") " pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:20.836779 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.836761    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbdk9\" (UniqueName: \"kubernetes.io/projected/2d26e486-242a-4a70-9c30-1481063fb109-kube-api-access-zbdk9\") pod \"e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") " pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:20.836891 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.836871    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-dshm\") pod \"e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") " pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:20.853585 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.853549    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:07:20.862288 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.862266    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-fade3a71-kserve-self-signed-certs\""
Nov 01 07:07:20.863962 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.863941    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"]
Nov 01 07:07:20.938252 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.938217    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-dshm\") pod \"e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") " pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:20.938470 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.938279    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-tmp-dir\") pod \"e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") " pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:20.938470 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.938297    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-model-cache\") pod \"e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") " pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:20.938470 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.938383    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-home\") pod \"e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") " pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:20.938470 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.938423    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d26e486-242a-4a70-9c30-1481063fb109-tls-certs\") pod \"e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") " pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:20.938702 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.938494    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbdk9\" (UniqueName: \"kubernetes.io/projected/2d26e486-242a-4a70-9c30-1481063fb109-kube-api-access-zbdk9\") pod \"e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") " pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:20.938815 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.938792    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-model-cache\") pod \"e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") " pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:20.938889 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.938826    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-home\") pod \"e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") " pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:20.938948 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.938893    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-tmp-dir\") pod \"e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") " pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:20.940542 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.940515    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-dshm\") pod \"e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") " pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:20.940802 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:20.940782    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d26e486-242a-4a70-9c30-1481063fb109-tls-certs\") pod \"e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") " pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:21.001042 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:21.001013    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbdk9\" (UniqueName: \"kubernetes.io/projected/2d26e486-242a-4a70-9c30-1481063fb109-kube-api-access-zbdk9\") pod \"e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") " pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:21.069624 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:21.069563    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:21.439504 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:21.439472    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"]
Nov 01 07:07:21.441231 ip-10-0-129-213 kubenswrapper[5619]: W1101 07:07:21.441207    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d26e486_242a_4a70_9c30_1481063fb109.slice/crio-45c4cdd6fbb8de106209b5d202cff55289fac8f44c8d02ef02378c05d65ae7fe WatchSource:0}: Error finding container 45c4cdd6fbb8de106209b5d202cff55289fac8f44c8d02ef02378c05d65ae7fe: Status 404 returned error can't find the container with id 45c4cdd6fbb8de106209b5d202cff55289fac8f44c8d02ef02378c05d65ae7fe
Nov 01 07:07:21.572514 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:21.572476    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:07:21.946152 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:21.946103    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z" event={"ID":"2d26e486-242a-4a70-9c30-1481063fb109","Type":"ContainerStarted","Data":"ce93d72774a1d4e9f60b20f5b0e89c93ba335de9eb028ac439f5399cb16bba26"}
Nov 01 07:07:21.946601 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:21.946164    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z" event={"ID":"2d26e486-242a-4a70-9c30-1481063fb109","Type":"ContainerStarted","Data":"45c4cdd6fbb8de106209b5d202cff55289fac8f44c8d02ef02378c05d65ae7fe"}
Nov 01 07:07:22.065266 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:22.065215    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z" podStartSLOduration=2.065200958 podStartE2EDuration="2.065200958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:07:22.063811859 +0000 UTC m=+4879.274635369" watchObservedRunningTime="2026-11-01 07:07:22.065200958 +0000 UTC m=+4879.276024491"
Nov 01 07:07:22.069906 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:22.069881    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:25.421431 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:25.421389    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:07:25.882445 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:07:25.882403    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:07:30.537021 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:30.536980    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:07:30.732558 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:30.732520    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:07:31.069797 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:31.069766    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:31.082357 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:31.082311    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:32.066037 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:32.066008    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:07:39.996946 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:39.996910    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:07:44.487754 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:07:44.487715    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:07:55.879055 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:07:55.879018    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:08:25.879051 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:08:25.879012    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:08:28.328422 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:28.328388    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"]
Nov 01 07:08:29.476517 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:29.476462    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:08:29.674805 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:29.674768    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:08:40.348411 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:40.348372    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:08:43.349145 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:43.349102    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z" podUID="2d26e486-242a-4a70-9c30-1481063fb109" containerName="main" containerID="cri-o://ce93d72774a1d4e9f60b20f5b0e89c93ba335de9eb028ac439f5399cb16bba26" gracePeriod=45
Nov 01 07:08:43.599421 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:43.599360    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:08:43.625680 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:43.625652    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d26e486-242a-4a70-9c30-1481063fb109-kube-api-access-zbdk9\" (UniqueName: \"kubernetes.io/projected/2d26e486-242a-4a70-9c30-1481063fb109-kube-api-access-zbdk9\") pod \"2d26e486-242a-4a70-9c30-1481063fb109\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") "
Nov 01 07:08:43.625800 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:43.625695    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-dshm\") pod \"2d26e486-242a-4a70-9c30-1481063fb109\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") "
Nov 01 07:08:43.625800 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:43.625718    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d26e486-242a-4a70-9c30-1481063fb109-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d26e486-242a-4a70-9c30-1481063fb109-tls-certs\") pod \"2d26e486-242a-4a70-9c30-1481063fb109\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") "
Nov 01 07:08:43.625800 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:43.625741    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-home\" (UniqueName: \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-home\") pod \"2d26e486-242a-4a70-9c30-1481063fb109\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") "
Nov 01 07:08:43.625921 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:43.625865    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-tmp-dir\") pod \"2d26e486-242a-4a70-9c30-1481063fb109\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") "
Nov 01 07:08:43.625979 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:43.625939    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-model-cache\") pod \"2d26e486-242a-4a70-9c30-1481063fb109\" (UID: \"2d26e486-242a-4a70-9c30-1481063fb109\") "
Nov 01 07:08:43.626151 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:43.626103    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-home" pod "2d26e486-242a-4a70-9c30-1481063fb109" (UID: "2d26e486-242a-4a70-9c30-1481063fb109"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:43.626227 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:43.626189    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-model-cache" pod "2d26e486-242a-4a70-9c30-1481063fb109" (UID: "2d26e486-242a-4a70-9c30-1481063fb109"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:43.626227 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:43.626202    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-tmp-dir" pod "2d26e486-242a-4a70-9c30-1481063fb109" (UID: "2d26e486-242a-4a70-9c30-1481063fb109"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:43.626391 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:43.626373    5619 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-tmp-dir\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:43.626454 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:43.626396    5619 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-model-cache\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:43.626454 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:43.626411    5619 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-home\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:43.627874 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:43.627852    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d26e486-242a-4a70-9c30-1481063fb109-kube-api-access-zbdk9" pod "2d26e486-242a-4a70-9c30-1481063fb109" (UID: "2d26e486-242a-4a70-9c30-1481063fb109"). InnerVolumeSpecName "kube-api-access-zbdk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:08:43.628220 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:43.628200    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d26e486-242a-4a70-9c30-1481063fb109-tls-certs" pod "2d26e486-242a-4a70-9c30-1481063fb109" (UID: "2d26e486-242a-4a70-9c30-1481063fb109"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:08:43.628288 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:43.628202    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-dshm" pod "2d26e486-242a-4a70-9c30-1481063fb109" (UID: "2d26e486-242a-4a70-9c30-1481063fb109"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:08:43.727110 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:43.727074    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbdk9\" (UniqueName: \"kubernetes.io/projected/2d26e486-242a-4a70-9c30-1481063fb109-kube-api-access-zbdk9\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:43.727110 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:43.727110    5619 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d26e486-242a-4a70-9c30-1481063fb109-dshm\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:43.727309 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:43.727124    5619 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d26e486-242a-4a70-9c30-1481063fb109-tls-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:08:44.447176 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:44.447140    5619 generic.go:356] "Generic (PLEG): container finished" podID="2d26e486-242a-4a70-9c30-1481063fb109" containerID="ce93d72774a1d4e9f60b20f5b0e89c93ba335de9eb028ac439f5399cb16bba26" exitCode=0
Nov 01 07:08:44.447670 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:44.447188    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"
Nov 01 07:08:44.447670 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:44.447223    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z" event={"ID":"2d26e486-242a-4a70-9c30-1481063fb109","Type":"ContainerDied","Data":"ce93d72774a1d4e9f60b20f5b0e89c93ba335de9eb028ac439f5399cb16bba26"}
Nov 01 07:08:44.447670 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:44.447269    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z" event={"ID":"2d26e486-242a-4a70-9c30-1481063fb109","Type":"ContainerDied","Data":"45c4cdd6fbb8de106209b5d202cff55289fac8f44c8d02ef02378c05d65ae7fe"}
Nov 01 07:08:44.447670 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:44.447285    5619 scope.go:122] "RemoveContainer" containerID="ce93d72774a1d4e9f60b20f5b0e89c93ba335de9eb028ac439f5399cb16bba26"
Nov 01 07:08:44.456238 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:44.456216    5619 scope.go:122] "RemoveContainer" containerID="ce93d72774a1d4e9f60b20f5b0e89c93ba335de9eb028ac439f5399cb16bba26"
Nov 01 07:08:44.456537 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:08:44.456515    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce93d72774a1d4e9f60b20f5b0e89c93ba335de9eb028ac439f5399cb16bba26\": container with ID starting with ce93d72774a1d4e9f60b20f5b0e89c93ba335de9eb028ac439f5399cb16bba26 not found: ID does not exist" containerID="ce93d72774a1d4e9f60b20f5b0e89c93ba335de9eb028ac439f5399cb16bba26"
Nov 01 07:08:44.456624 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:44.456547    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce93d72774a1d4e9f60b20f5b0e89c93ba335de9eb028ac439f5399cb16bba26"} err="failed to get container status \"ce93d72774a1d4e9f60b20f5b0e89c93ba335de9eb028ac439f5399cb16bba26\": rpc error: code = NotFound desc = could not find container \"ce93d72774a1d4e9f60b20f5b0e89c93ba335de9eb028ac439f5399cb16bba26\": container with ID starting with ce93d72774a1d4e9f60b20f5b0e89c93ba335de9eb028ac439f5399cb16bba26 not found: ID does not exist"
Nov 01 07:08:44.614258 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:44.614228    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"]
Nov 01 07:08:44.665148 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:44.665120    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-fade3a71-kserve-695fc9d96f-xht5z"]
Nov 01 07:08:44.831031 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:44.830927    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:08:44.963345 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:08:44.963280    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d26e486-242a-4a70-9c30-1481063fb109" path="/var/lib/kubelet/pods/2d26e486-242a-4a70-9c30-1481063fb109/volumes"
Nov 01 07:08:55.878611 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:08:55.878565    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:09:25.880125 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:09:25.880079    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:09:40.697267 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:40.697229    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:09:42.600751 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.600712    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"]
Nov 01 07:09:42.601136 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.601124    5619 cpu_manager.go:414] "containerMap: removing container" podUID="2d26e486-242a-4a70-9c30-1481063fb109" containerName="main"
Nov 01 07:09:42.601180 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.601135    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d26e486-242a-4a70-9c30-1481063fb109" containerName="main"
Nov 01 07:09:42.601216 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.601206    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d26e486-242a-4a70-9c30-1481063fb109" containerName="main"
Nov 01 07:09:42.604415 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.604398    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:42.646037 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.646007    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-tls-certs\") pod \"e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") " pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:42.646198 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.646086    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjrff\" (UniqueName: \"kubernetes.io/projected/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-kube-api-access-mjrff\") pod \"e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") " pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:42.646245 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.646223    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-home\") pod \"e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") " pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:42.646283 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.646264    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-tmp-dir\") pod \"e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") " pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:42.646394 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.646372    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-model-cache\") pod \"e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") " pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:42.646481 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.646404    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-dshm\") pod \"e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") " pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:42.678574 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.678542    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-8d2e6917-kserve-self-signed-certs\""
Nov 01 07:09:42.686878 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.686851    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:09:42.710506 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.710484    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"]
Nov 01 07:09:42.747339 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.747287    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-tls-certs\") pod \"e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") " pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:42.747596 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.747454    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjrff\" (UniqueName: \"kubernetes.io/projected/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-kube-api-access-mjrff\") pod \"e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") " pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:42.747596 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.747494    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-home\") pod \"e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") " pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:42.747596 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.747514    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-tmp-dir\") pod \"e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") " pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:42.747596 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.747539    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-model-cache\") pod \"e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") " pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:42.747596 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.747555    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-dshm\") pod \"e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") " pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:42.748184 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.748159    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-home\") pod \"e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") " pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:42.748394 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.748207    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-tmp-dir\") pod \"e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") " pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:42.748522 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.748489    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-model-cache\") pod \"e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") " pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:42.749997 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.749978    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-dshm\") pod \"e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") " pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:42.750998 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.750975    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-tls-certs\") pod \"e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") " pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:42.810006 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.809971    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjrff\" (UniqueName: \"kubernetes.io/projected/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-kube-api-access-mjrff\") pod \"e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") " pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:42.916313 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:42.916233    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:43.298099 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:43.298071    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"]
Nov 01 07:09:43.300467 ip-10-0-129-213 kubenswrapper[5619]: W1101 07:09:43.300439    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ff6a131_07f7_4af5_ba9b_3692701fb7ae.slice/crio-6cf7015d7aa809615130040c86be2e347b707e7127a3febd8be71e01b869d420 WatchSource:0}: Error finding container 6cf7015d7aa809615130040c86be2e347b707e7127a3febd8be71e01b869d420: Status 404 returned error can't find the container with id 6cf7015d7aa809615130040c86be2e347b707e7127a3febd8be71e01b869d420
Nov 01 07:09:43.739516 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:43.739483    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb" event={"ID":"2ff6a131-07f7-4af5-ba9b-3692701fb7ae","Type":"ContainerStarted","Data":"a9278a14a6dac93409325fd736044858f0fdb6d884682cdd96e43de411e4e021"}
Nov 01 07:09:43.739516 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:43.739520    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb" event={"ID":"2ff6a131-07f7-4af5-ba9b-3692701fb7ae","Type":"ContainerStarted","Data":"6cf7015d7aa809615130040c86be2e347b707e7127a3febd8be71e01b869d420"}
Nov 01 07:09:43.858966 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:43.858914    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb" podStartSLOduration=1.858901415 podStartE2EDuration="1.858901415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:09:43.858181795 +0000 UTC m=+5021.069005303" watchObservedRunningTime="2026-11-01 07:09:43.858901415 +0000 UTC m=+5021.069724928"
Nov 01 07:09:43.916815 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:43.916787    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:45.182023 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:45.181986    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:09:52.917139 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:52.917094    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:52.930072 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:52.930047    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:53.812424 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:09:53.812396    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:09:55.879235 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:09:55.879194    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:10:10.039131 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:10:10.039089    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:10:25.879674 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:10:25.879623    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:10:41.046158 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:10:41.046117    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:10:45.510516 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:10:45.510467    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:10:48.342557 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:10:48.342511    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-x9mzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:10:48.343067 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:10:48.342602    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj" podUID="d299e7a4-892c-404f-8346-91bcae60326e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:10:55.878858 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:10:55.878821    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:10:56.943558 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:10:56.943520    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"]
Nov 01 07:10:56.969744 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:10:56.969698    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:11:03.345566 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:03.345526    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-x9mzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:11:03.345972 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:03.345587    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x9mzj" podUID="d299e7a4-892c-404f-8346-91bcae60326e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:11:11.963286 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:11.963226    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb" podUID="2ff6a131-07f7-4af5-ba9b-3692701fb7ae" containerName="main" containerID="cri-o://a9278a14a6dac93409325fd736044858f0fdb6d884682cdd96e43de411e4e021" gracePeriod=45
Nov 01 07:11:12.215822 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.215762    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:11:12.238264 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.238236    5619 generic.go:356] "Generic (PLEG): container finished" podID="2ff6a131-07f7-4af5-ba9b-3692701fb7ae" containerID="a9278a14a6dac93409325fd736044858f0fdb6d884682cdd96e43de411e4e021" exitCode=0
Nov 01 07:11:12.238409 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.238279    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"
Nov 01 07:11:12.238409 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.238308    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb" event={"ID":"2ff6a131-07f7-4af5-ba9b-3692701fb7ae","Type":"ContainerDied","Data":"a9278a14a6dac93409325fd736044858f0fdb6d884682cdd96e43de411e4e021"}
Nov 01 07:11:12.238409 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.238371    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb" event={"ID":"2ff6a131-07f7-4af5-ba9b-3692701fb7ae","Type":"ContainerDied","Data":"6cf7015d7aa809615130040c86be2e347b707e7127a3febd8be71e01b869d420"}
Nov 01 07:11:12.238409 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.238395    5619 scope.go:122] "RemoveContainer" containerID="a9278a14a6dac93409325fd736044858f0fdb6d884682cdd96e43de411e4e021"
Nov 01 07:11:12.246541 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.246520    5619 scope.go:122] "RemoveContainer" containerID="a9278a14a6dac93409325fd736044858f0fdb6d884682cdd96e43de411e4e021"
Nov 01 07:11:12.246815 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:11:12.246795    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9278a14a6dac93409325fd736044858f0fdb6d884682cdd96e43de411e4e021\": container with ID starting with a9278a14a6dac93409325fd736044858f0fdb6d884682cdd96e43de411e4e021 not found: ID does not exist" containerID="a9278a14a6dac93409325fd736044858f0fdb6d884682cdd96e43de411e4e021"
Nov 01 07:11:12.246880 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.246824    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9278a14a6dac93409325fd736044858f0fdb6d884682cdd96e43de411e4e021"} err="failed to get container status \"a9278a14a6dac93409325fd736044858f0fdb6d884682cdd96e43de411e4e021\": rpc error: code = NotFound desc = could not find container \"a9278a14a6dac93409325fd736044858f0fdb6d884682cdd96e43de411e4e021\": container with ID starting with a9278a14a6dac93409325fd736044858f0fdb6d884682cdd96e43de411e4e021 not found: ID does not exist"
Nov 01 07:11:12.318393 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.318368    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-model-cache\") pod \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") "
Nov 01 07:11:12.318495 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.318405    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-dshm\") pod \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") "
Nov 01 07:11:12.318495 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.318443    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-home\" (UniqueName: \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-home\") pod \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") "
Nov 01 07:11:12.318495 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.318471    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-tls-certs\") pod \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") "
Nov 01 07:11:12.318667 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.318581    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-tmp-dir\") pod \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") "
Nov 01 07:11:12.318667 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.318618    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-kube-api-access-mjrff\" (UniqueName: \"kubernetes.io/projected/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-kube-api-access-mjrff\") pod \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\" (UID: \"2ff6a131-07f7-4af5-ba9b-3692701fb7ae\") "
Nov 01 07:11:12.318667 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.318649    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-model-cache" pod "2ff6a131-07f7-4af5-ba9b-3692701fb7ae" (UID: "2ff6a131-07f7-4af5-ba9b-3692701fb7ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:11:12.318827 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.318758    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-home" pod "2ff6a131-07f7-4af5-ba9b-3692701fb7ae" (UID: "2ff6a131-07f7-4af5-ba9b-3692701fb7ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:11:12.318883 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.318844    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-tmp-dir" pod "2ff6a131-07f7-4af5-ba9b-3692701fb7ae" (UID: "2ff6a131-07f7-4af5-ba9b-3692701fb7ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:11:12.319044 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.319026    5619 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-home\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:11:12.319105 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.319050    5619 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-tmp-dir\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:11:12.319105 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.319064    5619 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-model-cache\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:11:12.320739 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.320717    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-kube-api-access-mjrff" pod "2ff6a131-07f7-4af5-ba9b-3692701fb7ae" (UID: "2ff6a131-07f7-4af5-ba9b-3692701fb7ae"). InnerVolumeSpecName "kube-api-access-mjrff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:11:12.320926 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.320896    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-tls-certs" pod "2ff6a131-07f7-4af5-ba9b-3692701fb7ae" (UID: "2ff6a131-07f7-4af5-ba9b-3692701fb7ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:11:12.320926 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.320908    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-dshm" pod "2ff6a131-07f7-4af5-ba9b-3692701fb7ae" (UID: "2ff6a131-07f7-4af5-ba9b-3692701fb7ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:11:12.420579 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.420551    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjrff\" (UniqueName: \"kubernetes.io/projected/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-kube-api-access-mjrff\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:11:12.420579 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.420577    5619 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-dshm\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:11:12.420579 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.420587    5619 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ff6a131-07f7-4af5-ba9b-3692701fb7ae-tls-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:11:12.661203 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.661163    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"]
Nov 01 07:11:12.666605 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.666577    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-8d2e6917-kserve-66cf9995cc-twqvb"]
Nov 01 07:11:12.961608 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:12.961579    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff6a131-07f7-4af5-ba9b-3692701fb7ae" path="/var/lib/kubelet/pods/2ff6a131-07f7-4af5-ba9b-3692701fb7ae/volumes"
Nov 01 07:11:13.431146 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:13.431109    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:11:25.879875 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:11:25.879829    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:11:41.458665 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:41.458564    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:11:44.096177 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:44.096136    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:11:44.957557 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:44.957524    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:11:45.884432 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:45.884401    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:11:47.326102 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:47.326056    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:11:47.475713 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:47.475670    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:11:55.019565 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:55.019522    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr"]
Nov 01 07:11:55.020111 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:55.020090    5619 cpu_manager.go:414] "containerMap: removing container" podUID="2ff6a131-07f7-4af5-ba9b-3692701fb7ae" containerName="main"
Nov 01 07:11:55.020162 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:55.020112    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff6a131-07f7-4af5-ba9b-3692701fb7ae" containerName="main"
Nov 01 07:11:55.020199 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:55.020179    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff6a131-07f7-4af5-ba9b-3692701fb7ae" containerName="main"
Nov 01 07:11:55.023203 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:55.023187    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr"
Nov 01 07:11:55.084212 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:55.084174    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr"]
Nov 01 07:11:55.092002 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:55.091967    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bww8\" (UniqueName: \"kubernetes.io/projected/fbf9df6a-014c-427e-8b4e-8f9b0329797f-kube-api-access-6bww8\") pod \"payload-pre-processing-79f8d7f94f-pm7pr\" (UID: \"fbf9df6a-014c-427e-8b4e-8f9b0329797f\") " pod="openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr"
Nov 01 07:11:55.092206 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:55.092089    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbf9df6a-014c-427e-8b4e-8f9b0329797f-plugins-config-volume\") pod \"payload-pre-processing-79f8d7f94f-pm7pr\" (UID: \"fbf9df6a-014c-427e-8b4e-8f9b0329797f\") " pod="openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr"
Nov 01 07:11:55.193424 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:55.193389    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbf9df6a-014c-427e-8b4e-8f9b0329797f-plugins-config-volume\") pod \"payload-pre-processing-79f8d7f94f-pm7pr\" (UID: \"fbf9df6a-014c-427e-8b4e-8f9b0329797f\") " pod="openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr"
Nov 01 07:11:55.193619 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:55.193476    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bww8\" (UniqueName: \"kubernetes.io/projected/fbf9df6a-014c-427e-8b4e-8f9b0329797f-kube-api-access-6bww8\") pod \"payload-pre-processing-79f8d7f94f-pm7pr\" (UID: \"fbf9df6a-014c-427e-8b4e-8f9b0329797f\") " pod="openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr"
Nov 01 07:11:55.194122 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:55.194094    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbf9df6a-014c-427e-8b4e-8f9b0329797f-plugins-config-volume\") pod \"payload-pre-processing-79f8d7f94f-pm7pr\" (UID: \"fbf9df6a-014c-427e-8b4e-8f9b0329797f\") " pod="openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr"
Nov 01 07:11:55.255567 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:55.255528    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bww8\" (UniqueName: \"kubernetes.io/projected/fbf9df6a-014c-427e-8b4e-8f9b0329797f-kube-api-access-6bww8\") pod \"payload-pre-processing-79f8d7f94f-pm7pr\" (UID: \"fbf9df6a-014c-427e-8b4e-8f9b0329797f\") " pod="openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr"
Nov 01 07:11:55.337450 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:55.337354    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr"
Nov 01 07:11:55.707159 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:55.707090    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr"]
Nov 01 07:11:55.709409 ip-10-0-129-213 kubenswrapper[5619]: W1101 07:11:55.709380    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf9df6a_014c_427e_8b4e_8f9b0329797f.slice/crio-9e015ed54848d8e52b4a4fe05acf9eeef73249358614e9fe2ee4422bfd0cf983 WatchSource:0}: Error finding container 9e015ed54848d8e52b4a4fe05acf9eeef73249358614e9fe2ee4422bfd0cf983: Status 404 returned error can't find the container with id 9e015ed54848d8e52b4a4fe05acf9eeef73249358614e9fe2ee4422bfd0cf983
Nov 01 07:11:55.878804 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:11:55.878757    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:11:56.483543 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:56.483506    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr" event={"ID":"fbf9df6a-014c-427e-8b4e-8f9b0329797f","Type":"ContainerStarted","Data":"0bfd038258eba41822cec613123af3e1319feac60964d98b5741ffdc21bb8ca4"}
Nov 01 07:11:56.483933 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:56.483570    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr" event={"ID":"fbf9df6a-014c-427e-8b4e-8f9b0329797f","Type":"ContainerStarted","Data":"9e015ed54848d8e52b4a4fe05acf9eeef73249358614e9fe2ee4422bfd0cf983"}
Nov 01 07:11:56.483933 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:56.483718    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr"
Nov 01 07:11:56.601892 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:11:56.601839    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr" podStartSLOduration=2.601823345 podStartE2EDuration="2.601823345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:11:56.600556192 +0000 UTC m=+5153.811379738" watchObservedRunningTime="2026-11-01 07:11:56.601823345 +0000 UTC m=+5153.812646850"
Nov 01 07:12:07.491163 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:07.491134    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr"
Nov 01 07:12:09.698403 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:09.698368    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-bd6554cc7-b5tfg"]
Nov 01 07:12:09.703172 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:09.703151    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-bd6554cc7-b5tfg"
Nov 01 07:12:09.760820 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:09.760790    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-bd6554cc7-b5tfg"]
Nov 01 07:12:09.844640 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:09.844605    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blm6x\" (UniqueName: \"kubernetes.io/projected/f07b2451-892e-41fe-9b13-7eba91e3b0a3-kube-api-access-blm6x\") pod \"payload-processing-bd6554cc7-b5tfg\" (UID: \"f07b2451-892e-41fe-9b13-7eba91e3b0a3\") " pod="openshift-ingress/payload-processing-bd6554cc7-b5tfg"
Nov 01 07:12:09.844846 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:09.844721    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07b2451-892e-41fe-9b13-7eba91e3b0a3-plugins-config-volume\") pod \"payload-processing-bd6554cc7-b5tfg\" (UID: \"f07b2451-892e-41fe-9b13-7eba91e3b0a3\") " pod="openshift-ingress/payload-processing-bd6554cc7-b5tfg"
Nov 01 07:12:09.946292 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:09.946257    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blm6x\" (UniqueName: \"kubernetes.io/projected/f07b2451-892e-41fe-9b13-7eba91e3b0a3-kube-api-access-blm6x\") pod \"payload-processing-bd6554cc7-b5tfg\" (UID: \"f07b2451-892e-41fe-9b13-7eba91e3b0a3\") " pod="openshift-ingress/payload-processing-bd6554cc7-b5tfg"
Nov 01 07:12:09.946496 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:09.946364    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07b2451-892e-41fe-9b13-7eba91e3b0a3-plugins-config-volume\") pod \"payload-processing-bd6554cc7-b5tfg\" (UID: \"f07b2451-892e-41fe-9b13-7eba91e3b0a3\") " pod="openshift-ingress/payload-processing-bd6554cc7-b5tfg"
Nov 01 07:12:09.947031 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:09.947008    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07b2451-892e-41fe-9b13-7eba91e3b0a3-plugins-config-volume\") pod \"payload-processing-bd6554cc7-b5tfg\" (UID: \"f07b2451-892e-41fe-9b13-7eba91e3b0a3\") " pod="openshift-ingress/payload-processing-bd6554cc7-b5tfg"
Nov 01 07:12:10.008807 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:10.008735    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blm6x\" (UniqueName: \"kubernetes.io/projected/f07b2451-892e-41fe-9b13-7eba91e3b0a3-kube-api-access-blm6x\") pod \"payload-processing-bd6554cc7-b5tfg\" (UID: \"f07b2451-892e-41fe-9b13-7eba91e3b0a3\") " pod="openshift-ingress/payload-processing-bd6554cc7-b5tfg"
Nov 01 07:12:10.015419 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:10.015397    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-bd6554cc7-b5tfg"
Nov 01 07:12:10.388018 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:10.387988    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-bd6554cc7-b5tfg"]
Nov 01 07:12:10.389731 ip-10-0-129-213 kubenswrapper[5619]: W1101 07:12:10.389699    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07b2451_892e_41fe_9b13_7eba91e3b0a3.slice/crio-00347301c2922b8e02832485673f55494ea68a2e5c0c441cf189099a8fed348e WatchSource:0}: Error finding container 00347301c2922b8e02832485673f55494ea68a2e5c0c441cf189099a8fed348e: Status 404 returned error can't find the container with id 00347301c2922b8e02832485673f55494ea68a2e5c0c441cf189099a8fed348e
Nov 01 07:12:10.552104 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:10.552071    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-bd6554cc7-b5tfg" event={"ID":"f07b2451-892e-41fe-9b13-7eba91e3b0a3","Type":"ContainerStarted","Data":"c23508c16326e662a1a898dc30b776c195b8c326ba8c55681f1425d081ff307b"}
Nov 01 07:12:10.552249 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:10.552131    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-bd6554cc7-b5tfg" event={"ID":"f07b2451-892e-41fe-9b13-7eba91e3b0a3","Type":"ContainerStarted","Data":"00347301c2922b8e02832485673f55494ea68a2e5c0c441cf189099a8fed348e"}
Nov 01 07:12:10.552306 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:10.552262    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-bd6554cc7-b5tfg"
Nov 01 07:12:10.669824 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:10.669725    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-bd6554cc7-b5tfg" podStartSLOduration=1.669707785 podStartE2EDuration="1.669707785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:12:10.669452676 +0000 UTC m=+5167.880276187" watchObservedRunningTime="2026-11-01 07:12:10.669707785 +0000 UTC m=+5167.880531295"
Nov 01 07:12:21.560778 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:21.560748    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-bd6554cc7-b5tfg"
Nov 01 07:12:21.705366 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:21.705302    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-8444n"]
Nov 01 07:12:25.878264 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:12:25.878221    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:12:26.706537 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:26.706498    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-8444n" podUID="1f121a97-b0e6-422d-ac1b-f3a235b333e9" containerName="payload-processing" containerID="cri-o://a6e9c75b0971bcda5263381ee0650dd72ed140271d390169c6910811e2f30aea" gracePeriod=25
Nov 01 07:12:26.956122 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:26.956097    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-8444n"
Nov 01 07:12:27.018174 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:27.018085    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f121a97-b0e6-422d-ac1b-f3a235b333e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f121a97-b0e6-422d-ac1b-f3a235b333e9-plugins-config-volume\") pod \"1f121a97-b0e6-422d-ac1b-f3a235b333e9\" (UID: \"1f121a97-b0e6-422d-ac1b-f3a235b333e9\") "
Nov 01 07:12:27.018358 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:27.018207    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f121a97-b0e6-422d-ac1b-f3a235b333e9-kube-api-access-khtmv\" (UniqueName: \"kubernetes.io/projected/1f121a97-b0e6-422d-ac1b-f3a235b333e9-kube-api-access-khtmv\") pod \"1f121a97-b0e6-422d-ac1b-f3a235b333e9\" (UID: \"1f121a97-b0e6-422d-ac1b-f3a235b333e9\") "
Nov 01 07:12:27.018576 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:27.018550    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f121a97-b0e6-422d-ac1b-f3a235b333e9-plugins-config-volume" pod "1f121a97-b0e6-422d-ac1b-f3a235b333e9" (UID: "1f121a97-b0e6-422d-ac1b-f3a235b333e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:12:27.020380 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:27.020349    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f121a97-b0e6-422d-ac1b-f3a235b333e9-kube-api-access-khtmv" pod "1f121a97-b0e6-422d-ac1b-f3a235b333e9" (UID: "1f121a97-b0e6-422d-ac1b-f3a235b333e9"). InnerVolumeSpecName "kube-api-access-khtmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:12:27.119251 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:27.119220    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khtmv\" (UniqueName: \"kubernetes.io/projected/1f121a97-b0e6-422d-ac1b-f3a235b333e9-kube-api-access-khtmv\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:12:27.119251 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:27.119249    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f121a97-b0e6-422d-ac1b-f3a235b333e9-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:12:27.637980 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:27.637949    5619 generic.go:356] "Generic (PLEG): container finished" podID="1f121a97-b0e6-422d-ac1b-f3a235b333e9" containerID="a6e9c75b0971bcda5263381ee0650dd72ed140271d390169c6910811e2f30aea" exitCode=0
Nov 01 07:12:27.638177 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:27.638001    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-8444n"
Nov 01 07:12:27.638177 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:27.638040    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-8444n" event={"ID":"1f121a97-b0e6-422d-ac1b-f3a235b333e9","Type":"ContainerDied","Data":"a6e9c75b0971bcda5263381ee0650dd72ed140271d390169c6910811e2f30aea"}
Nov 01 07:12:27.638177 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:27.638096    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-8444n" event={"ID":"1f121a97-b0e6-422d-ac1b-f3a235b333e9","Type":"ContainerDied","Data":"db8dbc7ac66bd5cb54427ef23d595fc500a4bc38e8e00f59346e7ffbc0447023"}
Nov 01 07:12:27.638177 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:27.638125    5619 scope.go:122] "RemoveContainer" containerID="a6e9c75b0971bcda5263381ee0650dd72ed140271d390169c6910811e2f30aea"
Nov 01 07:12:27.646788 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:27.646768    5619 scope.go:122] "RemoveContainer" containerID="a6e9c75b0971bcda5263381ee0650dd72ed140271d390169c6910811e2f30aea"
Nov 01 07:12:27.647049 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:12:27.647029    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e9c75b0971bcda5263381ee0650dd72ed140271d390169c6910811e2f30aea\": container with ID starting with a6e9c75b0971bcda5263381ee0650dd72ed140271d390169c6910811e2f30aea not found: ID does not exist" containerID="a6e9c75b0971bcda5263381ee0650dd72ed140271d390169c6910811e2f30aea"
Nov 01 07:12:27.647106 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:27.647060    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e9c75b0971bcda5263381ee0650dd72ed140271d390169c6910811e2f30aea"} err="failed to get container status \"a6e9c75b0971bcda5263381ee0650dd72ed140271d390169c6910811e2f30aea\": rpc error: code = NotFound desc = could not find container \"a6e9c75b0971bcda5263381ee0650dd72ed140271d390169c6910811e2f30aea\": container with ID starting with a6e9c75b0971bcda5263381ee0650dd72ed140271d390169c6910811e2f30aea not found: ID does not exist"
Nov 01 07:12:27.804861 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:27.804827    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-8444n"]
Nov 01 07:12:27.808551 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:27.808530    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-8444n"]
Nov 01 07:12:28.961162 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:28.961120    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f121a97-b0e6-422d-ac1b-f3a235b333e9" path="/var/lib/kubelet/pods/1f121a97-b0e6-422d-ac1b-f3a235b333e9/volumes"
Nov 01 07:12:36.358275 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:36.358236    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-bd6554cc7-b5tfg"]
Nov 01 07:12:36.473391 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:36.473356    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr"]
Nov 01 07:12:41.359230 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.359171    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-bd6554cc7-b5tfg" podUID="f07b2451-892e-41fe-9b13-7eba91e3b0a3" containerName="payload-processing" containerID="cri-o://c23508c16326e662a1a898dc30b776c195b8c326ba8c55681f1425d081ff307b" gracePeriod=25
Nov 01 07:12:41.474317 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.474280    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr" podUID="fbf9df6a-014c-427e-8b4e-8f9b0329797f" containerName="payload-pre-processing" containerID="cri-o://0bfd038258eba41822cec613123af3e1319feac60964d98b5741ffdc21bb8ca4" gracePeriod=25
Nov 01 07:12:41.599668 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.599640    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-bd6554cc7-b5tfg"
Nov 01 07:12:41.711575 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.711542    5619 generic.go:356] "Generic (PLEG): container finished" podID="f07b2451-892e-41fe-9b13-7eba91e3b0a3" containerID="c23508c16326e662a1a898dc30b776c195b8c326ba8c55681f1425d081ff307b" exitCode=0
Nov 01 07:12:41.711732 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.711611    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-bd6554cc7-b5tfg"
Nov 01 07:12:41.711732 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.711620    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-bd6554cc7-b5tfg" event={"ID":"f07b2451-892e-41fe-9b13-7eba91e3b0a3","Type":"ContainerDied","Data":"c23508c16326e662a1a898dc30b776c195b8c326ba8c55681f1425d081ff307b"}
Nov 01 07:12:41.711862 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.711734    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-bd6554cc7-b5tfg" event={"ID":"f07b2451-892e-41fe-9b13-7eba91e3b0a3","Type":"ContainerDied","Data":"00347301c2922b8e02832485673f55494ea68a2e5c0c441cf189099a8fed348e"}
Nov 01 07:12:41.711862 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.711777    5619 scope.go:122] "RemoveContainer" containerID="c23508c16326e662a1a898dc30b776c195b8c326ba8c55681f1425d081ff307b"
Nov 01 07:12:41.713074 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.713052    5619 generic.go:356] "Generic (PLEG): container finished" podID="fbf9df6a-014c-427e-8b4e-8f9b0329797f" containerID="0bfd038258eba41822cec613123af3e1319feac60964d98b5741ffdc21bb8ca4" exitCode=0
Nov 01 07:12:41.713151 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.713123    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr" event={"ID":"fbf9df6a-014c-427e-8b4e-8f9b0329797f","Type":"ContainerDied","Data":"0bfd038258eba41822cec613123af3e1319feac60964d98b5741ffdc21bb8ca4"}
Nov 01 07:12:41.725829 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.725810    5619 scope.go:122] "RemoveContainer" containerID="c23508c16326e662a1a898dc30b776c195b8c326ba8c55681f1425d081ff307b"
Nov 01 07:12:41.726189 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:12:41.726118    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c23508c16326e662a1a898dc30b776c195b8c326ba8c55681f1425d081ff307b\": container with ID starting with c23508c16326e662a1a898dc30b776c195b8c326ba8c55681f1425d081ff307b not found: ID does not exist" containerID="c23508c16326e662a1a898dc30b776c195b8c326ba8c55681f1425d081ff307b"
Nov 01 07:12:41.726189 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.726156    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c23508c16326e662a1a898dc30b776c195b8c326ba8c55681f1425d081ff307b"} err="failed to get container status \"c23508c16326e662a1a898dc30b776c195b8c326ba8c55681f1425d081ff307b\": rpc error: code = NotFound desc = could not find container \"c23508c16326e662a1a898dc30b776c195b8c326ba8c55681f1425d081ff307b\": container with ID starting with c23508c16326e662a1a898dc30b776c195b8c326ba8c55681f1425d081ff307b not found: ID does not exist"
Nov 01 07:12:41.728430 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.728412    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr"
Nov 01 07:12:41.747977 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.747939    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:12:41.763831 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.763805    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f07b2451-892e-41fe-9b13-7eba91e3b0a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07b2451-892e-41fe-9b13-7eba91e3b0a3-plugins-config-volume\") pod \"f07b2451-892e-41fe-9b13-7eba91e3b0a3\" (UID: \"f07b2451-892e-41fe-9b13-7eba91e3b0a3\") "
Nov 01 07:12:41.763923 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.763895    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f07b2451-892e-41fe-9b13-7eba91e3b0a3-kube-api-access-blm6x\" (UniqueName: \"kubernetes.io/projected/f07b2451-892e-41fe-9b13-7eba91e3b0a3-kube-api-access-blm6x\") pod \"f07b2451-892e-41fe-9b13-7eba91e3b0a3\" (UID: \"f07b2451-892e-41fe-9b13-7eba91e3b0a3\") "
Nov 01 07:12:41.764142 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.764122    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07b2451-892e-41fe-9b13-7eba91e3b0a3-plugins-config-volume" pod "f07b2451-892e-41fe-9b13-7eba91e3b0a3" (UID: "f07b2451-892e-41fe-9b13-7eba91e3b0a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:12:41.764267 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.764247    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07b2451-892e-41fe-9b13-7eba91e3b0a3-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:12:41.765886 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.765868    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07b2451-892e-41fe-9b13-7eba91e3b0a3-kube-api-access-blm6x" pod "f07b2451-892e-41fe-9b13-7eba91e3b0a3" (UID: "f07b2451-892e-41fe-9b13-7eba91e3b0a3"). InnerVolumeSpecName "kube-api-access-blm6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:12:41.865318 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.865285    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbf9df6a-014c-427e-8b4e-8f9b0329797f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbf9df6a-014c-427e-8b4e-8f9b0329797f-plugins-config-volume\") pod \"fbf9df6a-014c-427e-8b4e-8f9b0329797f\" (UID: \"fbf9df6a-014c-427e-8b4e-8f9b0329797f\") "
Nov 01 07:12:41.865529 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.865366    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbf9df6a-014c-427e-8b4e-8f9b0329797f-kube-api-access-6bww8\" (UniqueName: \"kubernetes.io/projected/fbf9df6a-014c-427e-8b4e-8f9b0329797f-kube-api-access-6bww8\") pod \"fbf9df6a-014c-427e-8b4e-8f9b0329797f\" (UID: \"fbf9df6a-014c-427e-8b4e-8f9b0329797f\") "
Nov 01 07:12:41.865643 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.865621    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbf9df6a-014c-427e-8b4e-8f9b0329797f-plugins-config-volume" pod "fbf9df6a-014c-427e-8b4e-8f9b0329797f" (UID: "fbf9df6a-014c-427e-8b4e-8f9b0329797f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:12:41.865808 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.865794    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blm6x\" (UniqueName: \"kubernetes.io/projected/f07b2451-892e-41fe-9b13-7eba91e3b0a3-kube-api-access-blm6x\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:12:41.865851 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.865811    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbf9df6a-014c-427e-8b4e-8f9b0329797f-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:12:41.867283 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.867262    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf9df6a-014c-427e-8b4e-8f9b0329797f-kube-api-access-6bww8" pod "fbf9df6a-014c-427e-8b4e-8f9b0329797f" (UID: "fbf9df6a-014c-427e-8b4e-8f9b0329797f"). InnerVolumeSpecName "kube-api-access-6bww8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:12:41.966610 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:41.966533    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bww8\" (UniqueName: \"kubernetes.io/projected/fbf9df6a-014c-427e-8b4e-8f9b0329797f-kube-api-access-6bww8\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:12:42.133444 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:42.133405    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-bd6554cc7-b5tfg"]
Nov 01 07:12:42.137555 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:42.137527    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-bd6554cc7-b5tfg"]
Nov 01 07:12:42.559213 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:42.559170    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-bd6554cc7-b5tfg" podUID="f07b2451-892e-41fe-9b13-7eba91e3b0a3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.171:50052\" within 1s: context deadline exceeded"
Nov 01 07:12:42.559631 ip-10-0-129-213 kubenswrapper[5619]: W1101 07:12:42.559222    5619 logging.go:55] [core] [Channel #7788 SubChannel #7789]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.171:50052", ServerName: "10.133.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.171:50052: operation was canceled"
Nov 01 07:12:42.720431 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:42.720345    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr" event={"ID":"fbf9df6a-014c-427e-8b4e-8f9b0329797f","Type":"ContainerDied","Data":"9e015ed54848d8e52b4a4fe05acf9eeef73249358614e9fe2ee4422bfd0cf983"}
Nov 01 07:12:42.720431 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:42.720379    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr"
Nov 01 07:12:42.720431 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:42.720412    5619 scope.go:122] "RemoveContainer" containerID="0bfd038258eba41822cec613123af3e1319feac60964d98b5741ffdc21bb8ca4"
Nov 01 07:12:42.887830 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:42.887796    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr"]
Nov 01 07:12:42.892154 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:42.892131    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-79f8d7f94f-pm7pr"]
Nov 01 07:12:42.961549 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:42.961518    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f07b2451-892e-41fe-9b13-7eba91e3b0a3" path="/var/lib/kubelet/pods/f07b2451-892e-41fe-9b13-7eba91e3b0a3/volumes"
Nov 01 07:12:42.961927 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:42.961913    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbf9df6a-014c-427e-8b4e-8f9b0329797f" path="/var/lib/kubelet/pods/fbf9df6a-014c-427e-8b4e-8f9b0329797f/volumes"
Nov 01 07:12:46.226877 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:46.226836    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:12:49.019702 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:49.019671    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:12:49.464794 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:49.464760    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:12:52.086718 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:52.086682    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:12:53.255944 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.255904    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"]
Nov 01 07:12:53.256581 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.256559    5619 cpu_manager.go:414] "containerMap: removing container" podUID="1f121a97-b0e6-422d-ac1b-f3a235b333e9" containerName="payload-processing"
Nov 01 07:12:53.256632 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.256584    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f121a97-b0e6-422d-ac1b-f3a235b333e9" containerName="payload-processing"
Nov 01 07:12:53.256632 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.256625    5619 cpu_manager.go:414] "containerMap: removing container" podUID="f07b2451-892e-41fe-9b13-7eba91e3b0a3" containerName="payload-processing"
Nov 01 07:12:53.256704 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.256633    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07b2451-892e-41fe-9b13-7eba91e3b0a3" containerName="payload-processing"
Nov 01 07:12:53.256704 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.256649    5619 cpu_manager.go:414] "containerMap: removing container" podUID="fbf9df6a-014c-427e-8b4e-8f9b0329797f" containerName="payload-pre-processing"
Nov 01 07:12:53.256704 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.256658    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf9df6a-014c-427e-8b4e-8f9b0329797f" containerName="payload-pre-processing"
Nov 01 07:12:53.256801 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.256756    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f121a97-b0e6-422d-ac1b-f3a235b333e9" containerName="payload-processing"
Nov 01 07:12:53.256801 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.256773    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07b2451-892e-41fe-9b13-7eba91e3b0a3" containerName="payload-processing"
Nov 01 07:12:53.256801 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.256785    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf9df6a-014c-427e-8b4e-8f9b0329797f" containerName="payload-pre-processing"
Nov 01 07:12:53.260365 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.260345    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"
Nov 01 07:12:53.317663 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.317632    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"]
Nov 01 07:12:53.366542 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.366504    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\""
Nov 01 07:12:53.366542 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.366540    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 01 07:12:53.366793 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.366527    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 01 07:12:53.366793 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.366606    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 01 07:12:53.366793 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.366613    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 01 07:12:53.366793 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.366621    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-lpj49\""
Nov 01 07:12:53.374127 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.374077    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl54w\" (UniqueName: \"kubernetes.io/projected/3b3ce53b-28ef-4b90-a18b-cad00378db0a-kube-api-access-tl54w\") pod \"payload-processing-external-model-5bcd7997-qdl96\" (UID: \"3b3ce53b-28ef-4b90-a18b-cad00378db0a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"
Nov 01 07:12:53.374274 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.374136    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3ce53b-28ef-4b90-a18b-cad00378db0a-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-qdl96\" (UID: \"3b3ce53b-28ef-4b90-a18b-cad00378db0a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"
Nov 01 07:12:53.374373 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.374354    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b3ce53b-28ef-4b90-a18b-cad00378db0a-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-qdl96\" (UID: \"3b3ce53b-28ef-4b90-a18b-cad00378db0a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"
Nov 01 07:12:53.374482 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.374458    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b3ce53b-28ef-4b90-a18b-cad00378db0a-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-qdl96\" (UID: \"3b3ce53b-28ef-4b90-a18b-cad00378db0a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"
Nov 01 07:12:53.476012 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.475972    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b3ce53b-28ef-4b90-a18b-cad00378db0a-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-qdl96\" (UID: \"3b3ce53b-28ef-4b90-a18b-cad00378db0a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"
Nov 01 07:12:53.476195 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.476061    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b3ce53b-28ef-4b90-a18b-cad00378db0a-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-qdl96\" (UID: \"3b3ce53b-28ef-4b90-a18b-cad00378db0a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"
Nov 01 07:12:53.476195 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.476128    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl54w\" (UniqueName: \"kubernetes.io/projected/3b3ce53b-28ef-4b90-a18b-cad00378db0a-kube-api-access-tl54w\") pod \"payload-processing-external-model-5bcd7997-qdl96\" (UID: \"3b3ce53b-28ef-4b90-a18b-cad00378db0a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"
Nov 01 07:12:53.476342 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.476251    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3ce53b-28ef-4b90-a18b-cad00378db0a-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-qdl96\" (UID: \"3b3ce53b-28ef-4b90-a18b-cad00378db0a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"
Nov 01 07:12:53.476809 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.476782    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b3ce53b-28ef-4b90-a18b-cad00378db0a-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-qdl96\" (UID: \"3b3ce53b-28ef-4b90-a18b-cad00378db0a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"
Nov 01 07:12:53.476969 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.476951    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3ce53b-28ef-4b90-a18b-cad00378db0a-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-qdl96\" (UID: \"3b3ce53b-28ef-4b90-a18b-cad00378db0a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"
Nov 01 07:12:53.478476 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.478449    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b3ce53b-28ef-4b90-a18b-cad00378db0a-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-qdl96\" (UID: \"3b3ce53b-28ef-4b90-a18b-cad00378db0a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"
Nov 01 07:12:53.537900 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.537828    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl54w\" (UniqueName: \"kubernetes.io/projected/3b3ce53b-28ef-4b90-a18b-cad00378db0a-kube-api-access-tl54w\") pod \"payload-processing-external-model-5bcd7997-qdl96\" (UID: \"3b3ce53b-28ef-4b90-a18b-cad00378db0a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"
Nov 01 07:12:53.573890 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.573846    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"
Nov 01 07:12:53.948585 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:53.948543    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"]
Nov 01 07:12:53.950998 ip-10-0-129-213 kubenswrapper[5619]: W1101 07:12:53.950956    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b3ce53b_28ef_4b90_a18b_cad00378db0a.slice/crio-95a4427bfcf2607a911e5ae97642547bf43fe807773909de36b8a1965a2f1c0d WatchSource:0}: Error finding container 95a4427bfcf2607a911e5ae97642547bf43fe807773909de36b8a1965a2f1c0d: Status 404 returned error can't find the container with id 95a4427bfcf2607a911e5ae97642547bf43fe807773909de36b8a1965a2f1c0d
Nov 01 07:12:54.777646 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:54.777609    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96" event={"ID":"3b3ce53b-28ef-4b90-a18b-cad00378db0a","Type":"ContainerStarted","Data":"f695adb556cfef7599063bde5fd161581a01e92ed47e6651d28e7d4c00a8db71"}
Nov 01 07:12:54.778184 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:54.777667    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96" event={"ID":"3b3ce53b-28ef-4b90-a18b-cad00378db0a","Type":"ContainerStarted","Data":"95a4427bfcf2607a911e5ae97642547bf43fe807773909de36b8a1965a2f1c0d"}
Nov 01 07:12:54.778184 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:54.777765    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"
Nov 01 07:12:54.896438 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:12:54.896388    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96" podStartSLOduration=1.896373739 podStartE2EDuration="1.896373739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:12:54.895799061 +0000 UTC m=+5212.106622572" watchObservedRunningTime="2026-11-01 07:12:54.896373739 +0000 UTC m=+5212.107197245"
Nov 01 07:12:55.933820 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:12:55.933728    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:13:05.799589 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:13:05.799557    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"
Nov 01 07:13:25.879160 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:13:25.879120    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:13:42.104905 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:13:42.104863    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:13:46.626178 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:13:46.626142    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:13:55.878856 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:13:55.878818    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:14:25.878804 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:14:25.878767    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:14:42.466930 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:42.466845    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:14:45.259964 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:45.259928    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"]
Nov 01 07:14:45.408609 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:45.408572    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:14:45.598786 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:45.598750    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:14:45.928995 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:45.928880    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:14:46.920619 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:46.920578    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:14:50.261234 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:50.261153    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96" podUID="3b3ce53b-28ef-4b90-a18b-cad00378db0a" containerName="payload-processing" containerID="cri-o://f695adb556cfef7599063bde5fd161581a01e92ed47e6651d28e7d4c00a8db71" gracePeriod=25
Nov 01 07:14:50.546947 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:50.546922    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"
Nov 01 07:14:50.652809 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:50.652774    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b3ce53b-28ef-4b90-a18b-cad00378db0a-kube-api-access-tl54w\" (UniqueName: \"kubernetes.io/projected/3b3ce53b-28ef-4b90-a18b-cad00378db0a-kube-api-access-tl54w\") pod \"3b3ce53b-28ef-4b90-a18b-cad00378db0a\" (UID: \"3b3ce53b-28ef-4b90-a18b-cad00378db0a\") "
Nov 01 07:14:50.652809 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:50.652810    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b3ce53b-28ef-4b90-a18b-cad00378db0a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b3ce53b-28ef-4b90-a18b-cad00378db0a-provider-credentials\") pod \"3b3ce53b-28ef-4b90-a18b-cad00378db0a\" (UID: \"3b3ce53b-28ef-4b90-a18b-cad00378db0a\") "
Nov 01 07:14:50.653029 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:50.652846    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b3ce53b-28ef-4b90-a18b-cad00378db0a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b3ce53b-28ef-4b90-a18b-cad00378db0a-routing-overlay-volume\") pod \"3b3ce53b-28ef-4b90-a18b-cad00378db0a\" (UID: \"3b3ce53b-28ef-4b90-a18b-cad00378db0a\") "
Nov 01 07:14:50.653029 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:50.652968    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b3ce53b-28ef-4b90-a18b-cad00378db0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3ce53b-28ef-4b90-a18b-cad00378db0a-plugins-config-volume\") pod \"3b3ce53b-28ef-4b90-a18b-cad00378db0a\" (UID: \"3b3ce53b-28ef-4b90-a18b-cad00378db0a\") "
Nov 01 07:14:50.653282 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:50.653253    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b3ce53b-28ef-4b90-a18b-cad00378db0a-routing-overlay-volume" pod "3b3ce53b-28ef-4b90-a18b-cad00378db0a" (UID: "3b3ce53b-28ef-4b90-a18b-cad00378db0a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:14:50.653453 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:50.653380    5619 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b3ce53b-28ef-4b90-a18b-cad00378db0a-routing-overlay-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:50.653453 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:50.653430    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b3ce53b-28ef-4b90-a18b-cad00378db0a-plugins-config-volume" pod "3b3ce53b-28ef-4b90-a18b-cad00378db0a" (UID: "3b3ce53b-28ef-4b90-a18b-cad00378db0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:14:50.654954 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:50.654926    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b3ce53b-28ef-4b90-a18b-cad00378db0a-provider-credentials" pod "3b3ce53b-28ef-4b90-a18b-cad00378db0a" (UID: "3b3ce53b-28ef-4b90-a18b-cad00378db0a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:14:50.655080 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:50.654960    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b3ce53b-28ef-4b90-a18b-cad00378db0a-kube-api-access-tl54w" pod "3b3ce53b-28ef-4b90-a18b-cad00378db0a" (UID: "3b3ce53b-28ef-4b90-a18b-cad00378db0a"). InnerVolumeSpecName "kube-api-access-tl54w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:14:50.754569 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:50.754533    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3ce53b-28ef-4b90-a18b-cad00378db0a-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:50.754569 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:50.754562    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl54w\" (UniqueName: \"kubernetes.io/projected/3b3ce53b-28ef-4b90-a18b-cad00378db0a-kube-api-access-tl54w\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:50.754569 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:50.754572    5619 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b3ce53b-28ef-4b90-a18b-cad00378db0a-provider-credentials\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:14:51.338885 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:51.338793    5619 generic.go:356] "Generic (PLEG): container finished" podID="3b3ce53b-28ef-4b90-a18b-cad00378db0a" containerID="f695adb556cfef7599063bde5fd161581a01e92ed47e6651d28e7d4c00a8db71" exitCode=0
Nov 01 07:14:51.338885 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:51.338835    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"
Nov 01 07:14:51.339448 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:51.338879    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96" event={"ID":"3b3ce53b-28ef-4b90-a18b-cad00378db0a","Type":"ContainerDied","Data":"f695adb556cfef7599063bde5fd161581a01e92ed47e6651d28e7d4c00a8db71"}
Nov 01 07:14:51.339448 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:51.338941    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-qdl96" event={"ID":"3b3ce53b-28ef-4b90-a18b-cad00378db0a","Type":"ContainerDied","Data":"95a4427bfcf2607a911e5ae97642547bf43fe807773909de36b8a1965a2f1c0d"}
Nov 01 07:14:51.339448 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:51.338975    5619 scope.go:122] "RemoveContainer" containerID="f695adb556cfef7599063bde5fd161581a01e92ed47e6651d28e7d4c00a8db71"
Nov 01 07:14:51.347345 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:51.347309    5619 scope.go:122] "RemoveContainer" containerID="f695adb556cfef7599063bde5fd161581a01e92ed47e6651d28e7d4c00a8db71"
Nov 01 07:14:51.347628 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:14:51.347604    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f695adb556cfef7599063bde5fd161581a01e92ed47e6651d28e7d4c00a8db71\": container with ID starting with f695adb556cfef7599063bde5fd161581a01e92ed47e6651d28e7d4c00a8db71 not found: ID does not exist" containerID="f695adb556cfef7599063bde5fd161581a01e92ed47e6651d28e7d4c00a8db71"
Nov 01 07:14:51.347708 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:51.347640    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f695adb556cfef7599063bde5fd161581a01e92ed47e6651d28e7d4c00a8db71"} err="failed to get container status \"f695adb556cfef7599063bde5fd161581a01e92ed47e6651d28e7d4c00a8db71\": rpc error: code = NotFound desc = could not find container \"f695adb556cfef7599063bde5fd161581a01e92ed47e6651d28e7d4c00a8db71\": container with ID starting with f695adb556cfef7599063bde5fd161581a01e92ed47e6651d28e7d4c00a8db71 not found: ID does not exist"
Nov 01 07:14:51.505331 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:51.505299    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"]
Nov 01 07:14:51.556144 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:51.556113    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-qdl96"]
Nov 01 07:14:52.960592 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:14:52.960554    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b3ce53b-28ef-4b90-a18b-cad00378db0a" path="/var/lib/kubelet/pods/3b3ce53b-28ef-4b90-a18b-cad00378db0a/volumes"
Nov 01 07:14:55.878694 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:14:55.878650    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:15:25.878931 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:15:25.878888    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:15:42.801163 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:15:42.801127    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:15:47.256295 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:15:47.256244    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:15:55.938737 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:15:55.938699    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:16:00.964058 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:16:00.964019    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:16:25.885381 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:16:25.885342    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:16:43.197864 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:16:43.197820    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:16:47.621167 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:16:47.621116    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:16:55.938941 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:16:55.938893    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:17:25.879476 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:17:25.879437    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:17:43.504069 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:17:43.503983    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:17:47.982415 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:17:47.982379    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:17:55.878883 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:17:55.878844    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:18:25.878567 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:18:25.878530    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:18:43.847801 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:18:43.847758    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:18:48.328206 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:18:48.328161    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:18:55.879016 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:18:55.878974    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:19:25.878240 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:19:25.878202    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:19:44.225793 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:19:44.225745    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:19:48.680455 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:19:48.680412    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:19:55.882108 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:19:55.881993    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:20:25.879522 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:20:25.879484    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:20:44.554242 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:20:44.554148    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:20:49.005952 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:20:49.005905    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:20:55.878741 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:20:55.878706    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:21:12.145128 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.145092    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:12.658129 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.658084    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"]
Nov 01 07:21:12.658654 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.658632    5619 cpu_manager.go:414] "containerMap: removing container" podUID="3b3ce53b-28ef-4b90-a18b-cad00378db0a" containerName="payload-processing"
Nov 01 07:21:12.658783 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.658654    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b3ce53b-28ef-4b90-a18b-cad00378db0a" containerName="payload-processing"
Nov 01 07:21:12.658783 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.658751    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b3ce53b-28ef-4b90-a18b-cad00378db0a" containerName="payload-processing"
Nov 01 07:21:12.661808 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.661786    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.687568 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.687534    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-credential-socket\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.687718 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.687577    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-istio-data\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.687718 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.687632    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b330ed87-c300-412d-9ef5-91c4460e4794-istiod-ca-cert\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.687876 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.687842    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-workload-socket\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.688525 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.688473    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-istio-envoy\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.688718 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.688537    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-workload-certs\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.688718 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.688574    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txcrq\" (UniqueName: \"kubernetes.io/projected/b330ed87-c300-412d-9ef5-91c4460e4794-kube-api-access-txcrq\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.688718 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.688622    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b330ed87-c300-412d-9ef5-91c4460e4794-istio-token\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.688718 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.688683    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b330ed87-c300-412d-9ef5-91c4460e4794-istio-podinfo\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.716141 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.716107    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-c6c6bf-openshift-default-dockercfg-ml6gv\""
Nov 01 07:21:12.756743 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.756694    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:12.770246 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.770217    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"]
Nov 01 07:21:12.789315 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.789286    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-credential-socket\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.789470 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.789331    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-istio-data\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.789470 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.789381    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b330ed87-c300-412d-9ef5-91c4460e4794-istiod-ca-cert\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.789470 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.789410    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-workload-socket\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.789662 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.789544    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-istio-envoy\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.789662 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.789598    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-workload-certs\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.789662 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.789634    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txcrq\" (UniqueName: \"kubernetes.io/projected/b330ed87-c300-412d-9ef5-91c4460e4794-kube-api-access-txcrq\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.789832 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.789673    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b330ed87-c300-412d-9ef5-91c4460e4794-istio-token\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.789832 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.789716    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b330ed87-c300-412d-9ef5-91c4460e4794-istio-podinfo\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.789832 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.789732    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-credential-socket\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.790009 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.789843    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-workload-socket\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.790081 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.790023    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-workload-certs\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.790179 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.790159    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-istio-data\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.790302 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.790206    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b330ed87-c300-412d-9ef5-91c4460e4794-istiod-ca-cert\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.792306 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.792280    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b330ed87-c300-412d-9ef5-91c4460e4794-istio-podinfo\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.792446 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.792367    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-istio-envoy\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.852163 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.852129    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b330ed87-c300-412d-9ef5-91c4460e4794-istio-token\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.852372 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.852175    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txcrq\" (UniqueName: \"kubernetes.io/projected/b330ed87-c300-412d-9ef5-91c4460e4794-kube-api-access-txcrq\") pod \"e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") " pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:12.970804 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.970710    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:12.974482 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:12.974460    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:13.399553 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:13.399528    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"]
Nov 01 07:21:13.401620 ip-10-0-129-213 kubenswrapper[5619]: W1101 07:21:13.401594    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb330ed87_c300_412d_9ef5_91c4460e4794.slice/crio-46620e24b212098a9cd836355e9bb06f8469d06595413979cc1374a7c25103d5 WatchSource:0}: Error finding container 46620e24b212098a9cd836355e9bb06f8469d06595413979cc1374a7c25103d5: Status 404 returned error can't find the container with id 46620e24b212098a9cd836355e9bb06f8469d06595413979cc1374a7c25103d5
Nov 01 07:21:13.403667 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:13.403638    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 07:21:13.403727 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:13.403699    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 07:21:13.403767 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:13.403729    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 07:21:14.323204 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:14.323160    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96" event={"ID":"b330ed87-c300-412d-9ef5-91c4460e4794","Type":"ContainerStarted","Data":"7631a0dea019cf3c7b9afc91b27d471c50aaaade191953ca5adce1c098cdb81b"}
Nov 01 07:21:14.323432 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:14.323231    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96" event={"ID":"b330ed87-c300-412d-9ef5-91c4460e4794","Type":"ContainerStarted","Data":"46620e24b212098a9cd836355e9bb06f8469d06595413979cc1374a7c25103d5"}
Nov 01 07:21:14.446555 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:14.446498    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96" podStartSLOduration=2.446482545 podStartE2EDuration="2.446482545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:21:14.444944957 +0000 UTC m=+5711.655768465" watchObservedRunningTime="2026-11-01 07:21:14.446482545 +0000 UTC m=+5711.657306051"
Nov 01 07:21:14.975117 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:14.975086    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:14.980056 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:14.980034    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:15.328424 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:15.328339    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:15.329361 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:15.329341    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:17.344943 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:17.344907    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:17.545160 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:17.545125    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:17.693204 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:17.693111    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:20.593763 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:20.593726    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:20.787755 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:20.787722    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:21.001269 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.001183    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm"]
Nov 01 07:21:21.006233 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.006195    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm"
Nov 01 07:21:21.095346 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.072754    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fswjl\" (UniqueName: \"kubernetes.io/projected/615d0f5c-0f7a-4590-a0f7-41353672484a-kube-api-access-fswjl\") pod \"payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm\" (UID: \"615d0f5c-0f7a-4590-a0f7-41353672484a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm"
Nov 01 07:21:21.095346 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.072912    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615d0f5c-0f7a-4590-a0f7-41353672484a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm\" (UID: \"615d0f5c-0f7a-4590-a0f7-41353672484a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm"
Nov 01 07:21:21.113656 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.113571    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-c6c6bf\""
Nov 01 07:21:21.113656 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.113571    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-cb8e71dd-dockercfg-2g527\""
Nov 01 07:21:21.115514 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.115489    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm"]
Nov 01 07:21:21.149542 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.149515    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz"]
Nov 01 07:21:21.153136 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.153119    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz"
Nov 01 07:21:21.173627 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.173603    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fswjl\" (UniqueName: \"kubernetes.io/projected/615d0f5c-0f7a-4590-a0f7-41353672484a-kube-api-access-fswjl\") pod \"payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm\" (UID: \"615d0f5c-0f7a-4590-a0f7-41353672484a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm"
Nov 01 07:21:21.173741 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.173720    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615d0f5c-0f7a-4590-a0f7-41353672484a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm\" (UID: \"615d0f5c-0f7a-4590-a0f7-41353672484a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm"
Nov 01 07:21:21.174455 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.174430    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615d0f5c-0f7a-4590-a0f7-41353672484a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm\" (UID: \"615d0f5c-0f7a-4590-a0f7-41353672484a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm"
Nov 01 07:21:21.211840 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.211809    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz"]
Nov 01 07:21:21.235764 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.235732    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fswjl\" (UniqueName: \"kubernetes.io/projected/615d0f5c-0f7a-4590-a0f7-41353672484a-kube-api-access-fswjl\") pod \"payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm\" (UID: \"615d0f5c-0f7a-4590-a0f7-41353672484a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm"
Nov 01 07:21:21.274915 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.274844    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4tsb\" (UniqueName: \"kubernetes.io/projected/fa96b373-dd18-4bbe-babe-1a6d7cc2cb98-kube-api-access-z4tsb\") pod \"payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz\" (UID: \"fa96b373-dd18-4bbe-babe-1a6d7cc2cb98\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz"
Nov 01 07:21:21.275033 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.274977    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa96b373-dd18-4bbe-babe-1a6d7cc2cb98-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz\" (UID: \"fa96b373-dd18-4bbe-babe-1a6d7cc2cb98\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz"
Nov 01 07:21:21.321167 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.321142    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm"
Nov 01 07:21:21.377740 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.377706    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4tsb\" (UniqueName: \"kubernetes.io/projected/fa96b373-dd18-4bbe-babe-1a6d7cc2cb98-kube-api-access-z4tsb\") pod \"payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz\" (UID: \"fa96b373-dd18-4bbe-babe-1a6d7cc2cb98\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz"
Nov 01 07:21:21.377897 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.377877    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa96b373-dd18-4bbe-babe-1a6d7cc2cb98-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz\" (UID: \"fa96b373-dd18-4bbe-babe-1a6d7cc2cb98\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz"
Nov 01 07:21:21.378894 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.378866    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa96b373-dd18-4bbe-babe-1a6d7cc2cb98-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz\" (UID: \"fa96b373-dd18-4bbe-babe-1a6d7cc2cb98\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz"
Nov 01 07:21:21.441192 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.441156    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4tsb\" (UniqueName: \"kubernetes.io/projected/fa96b373-dd18-4bbe-babe-1a6d7cc2cb98-kube-api-access-z4tsb\") pod \"payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz\" (UID: \"fa96b373-dd18-4bbe-babe-1a6d7cc2cb98\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz"
Nov 01 07:21:21.466350 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.466301    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz"
Nov 01 07:21:21.693595 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.693564    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm"]
Nov 01 07:21:21.695088 ip-10-0-129-213 kubenswrapper[5619]: W1101 07:21:21.695048    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615d0f5c_0f7a_4590_a0f7_41353672484a.slice/crio-eb9ff6da22d981b36e1dc91961c263a60ed67555145930c5ee7c56685185b068 WatchSource:0}: Error finding container eb9ff6da22d981b36e1dc91961c263a60ed67555145930c5ee7c56685185b068: Status 404 returned error can't find the container with id eb9ff6da22d981b36e1dc91961c263a60ed67555145930c5ee7c56685185b068
Nov 01 07:21:21.845195 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:21.845171    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz"]
Nov 01 07:21:21.847258 ip-10-0-129-213 kubenswrapper[5619]: W1101 07:21:21.847221    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa96b373_dd18_4bbe_babe_1a6d7cc2cb98.slice/crio-1148c39ce8316bc93afc8f221fbe12b54a63a1fea9714c98a6ace0785c37bc3b WatchSource:0}: Error finding container 1148c39ce8316bc93afc8f221fbe12b54a63a1fea9714c98a6ace0785c37bc3b: Status 404 returned error can't find the container with id 1148c39ce8316bc93afc8f221fbe12b54a63a1fea9714c98a6ace0785c37bc3b
Nov 01 07:21:22.370854 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:22.370816    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz" event={"ID":"fa96b373-dd18-4bbe-babe-1a6d7cc2cb98","Type":"ContainerStarted","Data":"3689851e867fb3d931dbe8ee66ddf521d45949a09991c381d0d90520768602c5"}
Nov 01 07:21:22.371029 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:22.370876    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz" event={"ID":"fa96b373-dd18-4bbe-babe-1a6d7cc2cb98","Type":"ContainerStarted","Data":"1148c39ce8316bc93afc8f221fbe12b54a63a1fea9714c98a6ace0785c37bc3b"}
Nov 01 07:21:22.371029 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:22.370993    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz"
Nov 01 07:21:22.372490 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:22.372463    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm" event={"ID":"615d0f5c-0f7a-4590-a0f7-41353672484a","Type":"ContainerStarted","Data":"f3d44bbbf58bfcd3518173010a629d49d6bd6c1bb0679e88e522757c93e0bb45"}
Nov 01 07:21:22.372606 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:22.372506    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm" event={"ID":"615d0f5c-0f7a-4590-a0f7-41353672484a","Type":"ContainerStarted","Data":"eb9ff6da22d981b36e1dc91961c263a60ed67555145930c5ee7c56685185b068"}
Nov 01 07:21:22.372606 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:22.372586    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm"
Nov 01 07:21:22.488923 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:22.488871    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz" podStartSLOduration=1.488859923 podStartE2EDuration="1.488859923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:21:22.488616663 +0000 UTC m=+5719.699440194" watchObservedRunningTime="2026-11-01 07:21:22.488859923 +0000 UTC m=+5719.699683431"
Nov 01 07:21:22.615490 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:22.615423    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm" podStartSLOduration=2.615404153 podStartE2EDuration="2.615404153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:21:22.614465087 +0000 UTC m=+5719.825288596" watchObservedRunningTime="2026-11-01 07:21:22.615404153 +0000 UTC m=+5719.826227663"
Nov 01 07:21:23.592314 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:23.592269    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:23.788657 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:23.788620    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:23.933053 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:23.932958    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:25.878757 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:21:25.878722    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:21:28.472867 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:28.472825    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:33.297093 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:33.297049    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:33.380478 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:33.380446    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm"
Nov 01 07:21:33.380653 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:33.380514    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz"
Nov 01 07:21:38.117294 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:38.117253    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:41.468661 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:41.468576    5619 generic.go:356] "Generic (PLEG): container finished" podID="b330ed87-c300-412d-9ef5-91c4460e4794" containerID="7631a0dea019cf3c7b9afc91b27d471c50aaaade191953ca5adce1c098cdb81b" exitCode=0
Nov 01 07:21:41.468661 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:41.468648    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96" event={"ID":"b330ed87-c300-412d-9ef5-91c4460e4794","Type":"ContainerDied","Data":"7631a0dea019cf3c7b9afc91b27d471c50aaaade191953ca5adce1c098cdb81b"}
Nov 01 07:21:41.469117 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:41.469003    5619 scope.go:122] "RemoveContainer" containerID="7631a0dea019cf3c7b9afc91b27d471c50aaaade191953ca5adce1c098cdb81b"
Nov 01 07:21:41.470344 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:41.470290    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 07:21:41.470468 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:41.470394    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 07:21:41.470468 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:41.470445    5619 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 07:21:41.918125 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:41.918071    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:42.482669 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:42.482287    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96" event={"ID":"b330ed87-c300-412d-9ef5-91c4460e4794","Type":"ContainerStarted","Data":"cc255eca003a3ef3753ba00c2f2faaadef9a02c4a0ed2d926235a2cd6f5a4800"}
Nov 01 07:21:42.516672 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:42.516632    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:42.675976 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:42.675937    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:42.829146 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:42.829058    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:42.975580 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:42.975543    5619 patch_prober.go:29] interesting pod/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 01 07:21:42.975770 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:42.975600    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96" podUID="b330ed87-c300-412d-9ef5-91c4460e4794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 01 07:21:42.983025 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:42.982998    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:43.337205 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:43.337176    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:43.975873 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:43.975827    5619 patch_prober.go:29] interesting pod/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 01 07:21:43.976438 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:43.975883    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96" podUID="b330ed87-c300-412d-9ef5-91c4460e4794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 01 07:21:44.905454 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:44.905427    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:44.976339 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:44.975853    5619 patch_prober.go:29] interesting pod/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 01 07:21:44.976339 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:44.975924    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96" podUID="b330ed87-c300-412d-9ef5-91c4460e4794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 01 07:21:45.329073 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:45.328987    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:45.975751 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:45.975714    5619 patch_prober.go:29] interesting pod/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 01 07:21:45.975952 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:45.975770    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96" podUID="b330ed87-c300-412d-9ef5-91c4460e4794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 01 07:21:46.612678 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:46.612638    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:46.988463 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:46.986791    5619 patch_prober.go:29] interesting pod/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Nov 01 07:21:46.988463 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:46.986860    5619 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96" podUID="b330ed87-c300-412d-9ef5-91c4460e4794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Nov 01 07:21:47.979848 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:47.979815    5619 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:47.981041 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:47.981018    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:21:48.074251 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:48.074219    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:49.384099 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:49.384061    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:52.860966 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:52.860931    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:21:55.879293 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:21:55.879251    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:21:57.566419 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:21:57.566379    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:22:02.368267 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:02.368219    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:22:07.157516 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:07.157477    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:22:11.967733 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:11.967650    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:22:16.757142 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:16.757102    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:22:21.484140 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:21.484102    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:22:25.879261 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:22:25.879224    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:22:26.170804 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:26.170714    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:22:30.961929 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:30.961884    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:22:35.664261 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:35.664213    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:22:40.459590 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:40.459559    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:22:42.188154 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:42.188119    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:22:42.678554 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:42.678517    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:22:45.171257 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:45.171224    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:22:45.321593 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:45.321541    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:22:47.247783 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:47.247746    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:22:48.556355 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:48.556302    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:22:49.162067 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:49.162032    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz"]
Nov 01 07:22:49.269720 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:49.269689    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm"]
Nov 01 07:22:49.359614 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:49.359582    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:22:49.522315 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:49.522232    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:22:49.675425 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:49.675380    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:22:49.820433 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:49.820352    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:22:54.097685 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:54.097649    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:22:54.162970 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:54.162932    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz" podUID="fa96b373-dd18-4bbe-babe-1a6d7cc2cb98" containerName="payload-processing" containerID="cri-o://3689851e867fb3d931dbe8ee66ddf521d45949a09991c381d0d90520768602c5" gracePeriod=25
Nov 01 07:22:54.243842 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:54.243797    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:22:54.270545 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:22:54.270516    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm" podUID="615d0f5c-0f7a-4590-a0f7-41353672484a" containerName="payload-pre-processing" containerID="cri-o://f3d44bbbf58bfcd3518173010a629d49d6bd6c1bb0679e88e522757c93e0bb45" gracePeriod=25
Nov 01 07:22:55.878680 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:22:55.878641    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:23:03.378491 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:03.378449    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz" podUID="fa96b373-dd18-4bbe-babe-1a6d7cc2cb98" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 07:23:03.378967 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:03.378460    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm" podUID="615d0f5c-0f7a-4590-a0f7-41353672484a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 07:23:13.379243 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:13.379203    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm" podUID="615d0f5c-0f7a-4590-a0f7-41353672484a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 07:23:13.379647 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:13.379243    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz" podUID="fa96b373-dd18-4bbe-babe-1a6d7cc2cb98" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 07:23:14.417769 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.417709    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz"
Nov 01 07:23:14.510611 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.510584    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa96b373-dd18-4bbe-babe-1a6d7cc2cb98-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa96b373-dd18-4bbe-babe-1a6d7cc2cb98-plugins-config-volume\") pod \"fa96b373-dd18-4bbe-babe-1a6d7cc2cb98\" (UID: \"fa96b373-dd18-4bbe-babe-1a6d7cc2cb98\") "
Nov 01 07:23:14.510790 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.510773    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa96b373-dd18-4bbe-babe-1a6d7cc2cb98-kube-api-access-z4tsb\" (UniqueName: \"kubernetes.io/projected/fa96b373-dd18-4bbe-babe-1a6d7cc2cb98-kube-api-access-z4tsb\") pod \"fa96b373-dd18-4bbe-babe-1a6d7cc2cb98\" (UID: \"fa96b373-dd18-4bbe-babe-1a6d7cc2cb98\") "
Nov 01 07:23:14.511011 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.510982    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa96b373-dd18-4bbe-babe-1a6d7cc2cb98-plugins-config-volume" pod "fa96b373-dd18-4bbe-babe-1a6d7cc2cb98" (UID: "fa96b373-dd18-4bbe-babe-1a6d7cc2cb98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:23:14.511144 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.511127    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa96b373-dd18-4bbe-babe-1a6d7cc2cb98-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:23:14.512784 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.512760    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa96b373-dd18-4bbe-babe-1a6d7cc2cb98-kube-api-access-z4tsb" pod "fa96b373-dd18-4bbe-babe-1a6d7cc2cb98" (UID: "fa96b373-dd18-4bbe-babe-1a6d7cc2cb98"). InnerVolumeSpecName "kube-api-access-z4tsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:23:14.521731 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.521712    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm"
Nov 01 07:23:14.612479 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.612451    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/615d0f5c-0f7a-4590-a0f7-41353672484a-kube-api-access-fswjl\" (UniqueName: \"kubernetes.io/projected/615d0f5c-0f7a-4590-a0f7-41353672484a-kube-api-access-fswjl\") pod \"615d0f5c-0f7a-4590-a0f7-41353672484a\" (UID: \"615d0f5c-0f7a-4590-a0f7-41353672484a\") "
Nov 01 07:23:14.612651 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.612575    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/615d0f5c-0f7a-4590-a0f7-41353672484a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615d0f5c-0f7a-4590-a0f7-41353672484a-plugins-config-volume\") pod \"615d0f5c-0f7a-4590-a0f7-41353672484a\" (UID: \"615d0f5c-0f7a-4590-a0f7-41353672484a\") "
Nov 01 07:23:14.612834 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.612817    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4tsb\" (UniqueName: \"kubernetes.io/projected/fa96b373-dd18-4bbe-babe-1a6d7cc2cb98-kube-api-access-z4tsb\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:23:14.612945 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.612922    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/615d0f5c-0f7a-4590-a0f7-41353672484a-plugins-config-volume" pod "615d0f5c-0f7a-4590-a0f7-41353672484a" (UID: "615d0f5c-0f7a-4590-a0f7-41353672484a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:23:14.614502 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.614480    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615d0f5c-0f7a-4590-a0f7-41353672484a-kube-api-access-fswjl" pod "615d0f5c-0f7a-4590-a0f7-41353672484a" (UID: "615d0f5c-0f7a-4590-a0f7-41353672484a"). InnerVolumeSpecName "kube-api-access-fswjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:23:14.719039 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.718962    5619 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615d0f5c-0f7a-4590-a0f7-41353672484a-plugins-config-volume\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:23:14.719039 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.718990    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fswjl\" (UniqueName: \"kubernetes.io/projected/615d0f5c-0f7a-4590-a0f7-41353672484a-kube-api-access-fswjl\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:23:14.942437 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.942394    5619 generic.go:356] "Generic (PLEG): container finished" podID="615d0f5c-0f7a-4590-a0f7-41353672484a" containerID="f3d44bbbf58bfcd3518173010a629d49d6bd6c1bb0679e88e522757c93e0bb45" exitCode=0
Nov 01 07:23:14.942610 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.942461    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm"
Nov 01 07:23:14.942610 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.942483    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm" event={"ID":"615d0f5c-0f7a-4590-a0f7-41353672484a","Type":"ContainerDied","Data":"f3d44bbbf58bfcd3518173010a629d49d6bd6c1bb0679e88e522757c93e0bb45"}
Nov 01 07:23:14.942610 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.942582    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm" event={"ID":"615d0f5c-0f7a-4590-a0f7-41353672484a","Type":"ContainerDied","Data":"eb9ff6da22d981b36e1dc91961c263a60ed67555145930c5ee7c56685185b068"}
Nov 01 07:23:14.942610 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.942610    5619 scope.go:122] "RemoveContainer" containerID="f3d44bbbf58bfcd3518173010a629d49d6bd6c1bb0679e88e522757c93e0bb45"
Nov 01 07:23:14.943758 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.943732    5619 generic.go:356] "Generic (PLEG): container finished" podID="fa96b373-dd18-4bbe-babe-1a6d7cc2cb98" containerID="3689851e867fb3d931dbe8ee66ddf521d45949a09991c381d0d90520768602c5" exitCode=0
Nov 01 07:23:14.943847 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.943783    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz"
Nov 01 07:23:14.943912 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.943774    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz" event={"ID":"fa96b373-dd18-4bbe-babe-1a6d7cc2cb98","Type":"ContainerDied","Data":"3689851e867fb3d931dbe8ee66ddf521d45949a09991c381d0d90520768602c5"}
Nov 01 07:23:14.943912 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.943898    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz" event={"ID":"fa96b373-dd18-4bbe-babe-1a6d7cc2cb98","Type":"ContainerDied","Data":"1148c39ce8316bc93afc8f221fbe12b54a63a1fea9714c98a6ace0785c37bc3b"}
Nov 01 07:23:14.952952 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.952929    5619 scope.go:122] "RemoveContainer" containerID="f3d44bbbf58bfcd3518173010a629d49d6bd6c1bb0679e88e522757c93e0bb45"
Nov 01 07:23:14.953192 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:23:14.953172    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3d44bbbf58bfcd3518173010a629d49d6bd6c1bb0679e88e522757c93e0bb45\": container with ID starting with f3d44bbbf58bfcd3518173010a629d49d6bd6c1bb0679e88e522757c93e0bb45 not found: ID does not exist" containerID="f3d44bbbf58bfcd3518173010a629d49d6bd6c1bb0679e88e522757c93e0bb45"
Nov 01 07:23:14.953263 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.953199    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3d44bbbf58bfcd3518173010a629d49d6bd6c1bb0679e88e522757c93e0bb45"} err="failed to get container status \"f3d44bbbf58bfcd3518173010a629d49d6bd6c1bb0679e88e522757c93e0bb45\": rpc error: code = NotFound desc = could not find container \"f3d44bbbf58bfcd3518173010a629d49d6bd6c1bb0679e88e522757c93e0bb45\": container with ID starting with f3d44bbbf58bfcd3518173010a629d49d6bd6c1bb0679e88e522757c93e0bb45 not found: ID does not exist"
Nov 01 07:23:14.953263 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.953222    5619 scope.go:122] "RemoveContainer" containerID="3689851e867fb3d931dbe8ee66ddf521d45949a09991c381d0d90520768602c5"
Nov 01 07:23:14.962488 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.962470    5619 scope.go:122] "RemoveContainer" containerID="3689851e867fb3d931dbe8ee66ddf521d45949a09991c381d0d90520768602c5"
Nov 01 07:23:14.962756 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:23:14.962731    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3689851e867fb3d931dbe8ee66ddf521d45949a09991c381d0d90520768602c5\": container with ID starting with 3689851e867fb3d931dbe8ee66ddf521d45949a09991c381d0d90520768602c5 not found: ID does not exist" containerID="3689851e867fb3d931dbe8ee66ddf521d45949a09991c381d0d90520768602c5"
Nov 01 07:23:14.962836 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:14.962767    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3689851e867fb3d931dbe8ee66ddf521d45949a09991c381d0d90520768602c5"} err="failed to get container status \"3689851e867fb3d931dbe8ee66ddf521d45949a09991c381d0d90520768602c5\": rpc error: code = NotFound desc = could not find container \"3689851e867fb3d931dbe8ee66ddf521d45949a09991c381d0d90520768602c5\": container with ID starting with 3689851e867fb3d931dbe8ee66ddf521d45949a09991c381d0d90520768602c5 not found: ID does not exist"
Nov 01 07:23:15.112231 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:15.112196    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm"]
Nov 01 07:23:15.162838 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:15.162807    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c6c6bf-655cffbkqm"]
Nov 01 07:23:15.227947 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:15.227909    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz"]
Nov 01 07:23:15.232143 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:15.232115    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c6c6bf-5c5b8b956k52zz"]
Nov 01 07:23:16.960910 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:16.960880    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="615d0f5c-0f7a-4590-a0f7-41353672484a" path="/var/lib/kubelet/pods/615d0f5c-0f7a-4590-a0f7-41353672484a/volumes"
Nov 01 07:23:16.961342 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:16.961302    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa96b373-dd18-4bbe-babe-1a6d7cc2cb98" path="/var/lib/kubelet/pods/fa96b373-dd18-4bbe-babe-1a6d7cc2cb98/volumes"
Nov 01 07:23:23.770588 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:23.770551    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:23:23.922863 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:23.922818    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:23:24.495654 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:24.495615    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:23:24.666963 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:24.666904    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:23:25.728163 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:25.728127    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"]
Nov 01 07:23:25.728603 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:25.728588    5619 cpu_manager.go:414] "containerMap: removing container" podUID="615d0f5c-0f7a-4590-a0f7-41353672484a" containerName="payload-pre-processing"
Nov 01 07:23:25.728651 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:25.728603    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="615d0f5c-0f7a-4590-a0f7-41353672484a" containerName="payload-pre-processing"
Nov 01 07:23:25.728651 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:25.728625    5619 cpu_manager.go:414] "containerMap: removing container" podUID="fa96b373-dd18-4bbe-babe-1a6d7cc2cb98" containerName="payload-processing"
Nov 01 07:23:25.728651 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:25.728630    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa96b373-dd18-4bbe-babe-1a6d7cc2cb98" containerName="payload-processing"
Nov 01 07:23:25.728756 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:25.728698    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="615d0f5c-0f7a-4590-a0f7-41353672484a" containerName="payload-pre-processing"
Nov 01 07:23:25.728756 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:25.728709    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa96b373-dd18-4bbe-babe-1a6d7cc2cb98" containerName="payload-processing"
Nov 01 07:23:25.731623 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:25.731606    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"
Nov 01 07:23:25.785016 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:25.784988    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-c6c6bf-serving-cert\""
Nov 01 07:23:25.827610 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:25.827579    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc0309f-a41e-4497-a576-3422db4b8ba9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p\" (UID: \"7cc0309f-a41e-4497-a576-3422db4b8ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"
Nov 01 07:23:25.827813 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:25.827633    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cc0309f-a41e-4497-a576-3422db4b8ba9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p\" (UID: \"7cc0309f-a41e-4497-a576-3422db4b8ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"
Nov 01 07:23:25.827813 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:25.827712    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cc0309f-a41e-4497-a576-3422db4b8ba9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p\" (UID: \"7cc0309f-a41e-4497-a576-3422db4b8ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"
Nov 01 07:23:25.827813 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:25.827805    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzkc4\" (UniqueName: \"kubernetes.io/projected/7cc0309f-a41e-4497-a576-3422db4b8ba9-kube-api-access-tzkc4\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p\" (UID: \"7cc0309f-a41e-4497-a576-3422db4b8ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"
Nov 01 07:23:25.837588 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:25.837556    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"]
Nov 01 07:23:25.879727 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:23:25.879687    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:23:25.920638 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:25.920600    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:23:25.928739 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:25.928712    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzkc4\" (UniqueName: \"kubernetes.io/projected/7cc0309f-a41e-4497-a576-3422db4b8ba9-kube-api-access-tzkc4\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p\" (UID: \"7cc0309f-a41e-4497-a576-3422db4b8ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"
Nov 01 07:23:25.928834 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:25.928805    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc0309f-a41e-4497-a576-3422db4b8ba9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p\" (UID: \"7cc0309f-a41e-4497-a576-3422db4b8ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"
Nov 01 07:23:25.928878 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:25.928857    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cc0309f-a41e-4497-a576-3422db4b8ba9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p\" (UID: \"7cc0309f-a41e-4497-a576-3422db4b8ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"
Nov 01 07:23:25.928919 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:25.928903    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cc0309f-a41e-4497-a576-3422db4b8ba9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p\" (UID: \"7cc0309f-a41e-4497-a576-3422db4b8ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"
Nov 01 07:23:25.929034 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:23:25.929011    5619 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-serving-cert: secret "maas-api-e2e-worker-single-main-c6c6bf-serving-cert" not found
Nov 01 07:23:25.929098 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:23:25.929088    5619 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc0309f-a41e-4497-a576-3422db4b8ba9-maas-api-tls podName:7cc0309f-a41e-4497-a576-3422db4b8ba9 nodeName:}" failed. No retries permitted until 2026-11-01 07:23:26.429072248 +0000 UTC m=+5843.639895736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7cc0309f-a41e-4497-a576-3422db4b8ba9-maas-api-tls") pod "maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p" (UID: "7cc0309f-a41e-4497-a576-3422db4b8ba9") : secret "maas-api-e2e-worker-single-main-c6c6bf-serving-cert" not found
Nov 01 07:23:25.929497 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:25.929475    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc0309f-a41e-4497-a576-3422db4b8ba9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p\" (UID: \"7cc0309f-a41e-4497-a576-3422db4b8ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"
Nov 01 07:23:25.931304 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:25.931280    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cc0309f-a41e-4497-a576-3422db4b8ba9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p\" (UID: \"7cc0309f-a41e-4497-a576-3422db4b8ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"
Nov 01 07:23:25.991134 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:25.991065    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzkc4\" (UniqueName: \"kubernetes.io/projected/7cc0309f-a41e-4497-a576-3422db4b8ba9-kube-api-access-tzkc4\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p\" (UID: \"7cc0309f-a41e-4497-a576-3422db4b8ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"
Nov 01 07:23:26.113306 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:26.113273    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:23:26.434622 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:26.434571    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cc0309f-a41e-4497-a576-3422db4b8ba9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p\" (UID: \"7cc0309f-a41e-4497-a576-3422db4b8ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"
Nov 01 07:23:26.437211 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:26.437183    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cc0309f-a41e-4497-a576-3422db4b8ba9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p\" (UID: \"7cc0309f-a41e-4497-a576-3422db4b8ba9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"
Nov 01 07:23:26.645155 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:26.645119    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"
Nov 01 07:23:27.028129 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:27.028107    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"]
Nov 01 07:23:27.029887 ip-10-0-129-213 kubenswrapper[5619]: W1101 07:23:27.029851    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc0309f_a41e_4497_a576_3422db4b8ba9.slice/crio-c23580fc44a61b5e91162303556f811df487098eae072e77d47a2acdf248aa8d WatchSource:0}: Error finding container c23580fc44a61b5e91162303556f811df487098eae072e77d47a2acdf248aa8d: Status 404 returned error can't find the container with id c23580fc44a61b5e91162303556f811df487098eae072e77d47a2acdf248aa8d
Nov 01 07:23:27.031262 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:27.031243    5619 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 07:23:28.011269 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:28.011233    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p" event={"ID":"7cc0309f-a41e-4497-a576-3422db4b8ba9","Type":"ContainerStarted","Data":"c23580fc44a61b5e91162303556f811df487098eae072e77d47a2acdf248aa8d"}
Nov 01 07:23:29.018388 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:29.018347    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p" event={"ID":"7cc0309f-a41e-4497-a576-3422db4b8ba9","Type":"ContainerStarted","Data":"d4c687af28ea7b36277c99b89e3c36375291e4ad32637ebecc8ec8fbbea28c95"}
Nov 01 07:23:29.018804 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:29.018410    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"
Nov 01 07:23:29.163760 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:29.163708    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p" podStartSLOduration=3.165712251 podStartE2EDuration="4.163691727s" totalImagesPullingTime="997.979476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:23:29.162582978 +0000 UTC m=+5846.373406488" watchObservedRunningTime="2026-11-01 07:23:29.163691727 +0000 UTC m=+5846.374515233"
Nov 01 07:23:30.543970 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:30.543931    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:23:30.744128 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:30.744094    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:23:30.890280 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:30.890244    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:23:35.028475 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:35.028439    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"
Nov 01 07:23:35.582683 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:35.582648    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:23:40.258551 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:40.258507    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:23:45.072089 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:45.071868    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:23:45.600839 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:45.600791    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:23:49.779974 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:49.779938    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:23:50.080679 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:50.080585    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:23:53.814406 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:53.814374    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"]
Nov 01 07:23:53.814858 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:53.814590    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p" podUID="7cc0309f-a41e-4497-a576-3422db4b8ba9" containerName="maas-api" containerID="cri-o://d4c687af28ea7b36277c99b89e3c36375291e4ad32637ebecc8ec8fbbea28c95" gracePeriod=30
Nov 01 07:23:53.919605 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:53.919568    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:23:54.071418 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.071349    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"
Nov 01 07:23:54.137615 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.137576    5619 generic.go:356] "Generic (PLEG): container finished" podID="7cc0309f-a41e-4497-a576-3422db4b8ba9" containerID="d4c687af28ea7b36277c99b89e3c36375291e4ad32637ebecc8ec8fbbea28c95" exitCode=0
Nov 01 07:23:54.137790 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.137624    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"
Nov 01 07:23:54.137790 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.137667    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p" event={"ID":"7cc0309f-a41e-4497-a576-3422db4b8ba9","Type":"ContainerDied","Data":"d4c687af28ea7b36277c99b89e3c36375291e4ad32637ebecc8ec8fbbea28c95"}
Nov 01 07:23:54.137790 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.137730    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p" event={"ID":"7cc0309f-a41e-4497-a576-3422db4b8ba9","Type":"ContainerDied","Data":"c23580fc44a61b5e91162303556f811df487098eae072e77d47a2acdf248aa8d"}
Nov 01 07:23:54.137790 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.137761    5619 scope.go:122] "RemoveContainer" containerID="d4c687af28ea7b36277c99b89e3c36375291e4ad32637ebecc8ec8fbbea28c95"
Nov 01 07:23:54.146379 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.146350    5619 scope.go:122] "RemoveContainer" containerID="d4c687af28ea7b36277c99b89e3c36375291e4ad32637ebecc8ec8fbbea28c95"
Nov 01 07:23:54.146647 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:23:54.146624    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4c687af28ea7b36277c99b89e3c36375291e4ad32637ebecc8ec8fbbea28c95\": container with ID starting with d4c687af28ea7b36277c99b89e3c36375291e4ad32637ebecc8ec8fbbea28c95 not found: ID does not exist" containerID="d4c687af28ea7b36277c99b89e3c36375291e4ad32637ebecc8ec8fbbea28c95"
Nov 01 07:23:54.146715 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.146660    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4c687af28ea7b36277c99b89e3c36375291e4ad32637ebecc8ec8fbbea28c95"} err="failed to get container status \"d4c687af28ea7b36277c99b89e3c36375291e4ad32637ebecc8ec8fbbea28c95\": rpc error: code = NotFound desc = could not find container \"d4c687af28ea7b36277c99b89e3c36375291e4ad32637ebecc8ec8fbbea28c95\": container with ID starting with d4c687af28ea7b36277c99b89e3c36375291e4ad32637ebecc8ec8fbbea28c95 not found: ID does not exist"
Nov 01 07:23:54.214823 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.214792    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cc0309f-a41e-4497-a576-3422db4b8ba9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc0309f-a41e-4497-a576-3422db4b8ba9-openshift-service-ca\") pod \"7cc0309f-a41e-4497-a576-3422db4b8ba9\" (UID: \"7cc0309f-a41e-4497-a576-3422db4b8ba9\") "
Nov 01 07:23:54.215024 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.214834    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cc0309f-a41e-4497-a576-3422db4b8ba9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cc0309f-a41e-4497-a576-3422db4b8ba9-metrics-certs\") pod \"7cc0309f-a41e-4497-a576-3422db4b8ba9\" (UID: \"7cc0309f-a41e-4497-a576-3422db4b8ba9\") "
Nov 01 07:23:54.215024 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.214878    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cc0309f-a41e-4497-a576-3422db4b8ba9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cc0309f-a41e-4497-a576-3422db4b8ba9-maas-api-tls\") pod \"7cc0309f-a41e-4497-a576-3422db4b8ba9\" (UID: \"7cc0309f-a41e-4497-a576-3422db4b8ba9\") "
Nov 01 07:23:54.215257 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.215230    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc0309f-a41e-4497-a576-3422db4b8ba9-openshift-service-ca" pod "7cc0309f-a41e-4497-a576-3422db4b8ba9" (UID: "7cc0309f-a41e-4497-a576-3422db4b8ba9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:23:54.215452 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.215431    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cc0309f-a41e-4497-a576-3422db4b8ba9-kube-api-access-tzkc4\" (UniqueName: \"kubernetes.io/projected/7cc0309f-a41e-4497-a576-3422db4b8ba9-kube-api-access-tzkc4\") pod \"7cc0309f-a41e-4497-a576-3422db4b8ba9\" (UID: \"7cc0309f-a41e-4497-a576-3422db4b8ba9\") "
Nov 01 07:23:54.215812 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.215792    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc0309f-a41e-4497-a576-3422db4b8ba9-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:23:54.218864 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.218011    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cc0309f-a41e-4497-a576-3422db4b8ba9-maas-api-tls" pod "7cc0309f-a41e-4497-a576-3422db4b8ba9" (UID: "7cc0309f-a41e-4497-a576-3422db4b8ba9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:23:54.219229 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.219167    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc0309f-a41e-4497-a576-3422db4b8ba9-kube-api-access-tzkc4" pod "7cc0309f-a41e-4497-a576-3422db4b8ba9" (UID: "7cc0309f-a41e-4497-a576-3422db4b8ba9"). InnerVolumeSpecName "kube-api-access-tzkc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:23:54.219580 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.219530    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cc0309f-a41e-4497-a576-3422db4b8ba9-metrics-certs" pod "7cc0309f-a41e-4497-a576-3422db4b8ba9" (UID: "7cc0309f-a41e-4497-a576-3422db4b8ba9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:23:54.316652 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.316620    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cc0309f-a41e-4497-a576-3422db4b8ba9-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:23:54.316652 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.316649    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzkc4\" (UniqueName: \"kubernetes.io/projected/7cc0309f-a41e-4497-a576-3422db4b8ba9-kube-api-access-tzkc4\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:23:54.316862 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.316661    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cc0309f-a41e-4497-a576-3422db4b8ba9-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:23:54.562019 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.561983    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"]
Nov 01 07:23:54.565754 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.565729    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c6c6bf-78fd4f48fd-vqx9p"]
Nov 01 07:23:54.961906 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:54.961860    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cc0309f-a41e-4497-a576-3422db4b8ba9" path="/var/lib/kubelet/pods/7cc0309f-a41e-4497-a576-3422db4b8ba9/volumes"
Nov 01 07:23:55.439972 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:55.439935    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:23:55.710698 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:55.710605    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:23:55.879820 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:23:55.879774    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:23:56.338837 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:56.338806    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"]
Nov 01 07:23:56.339312 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:56.339040    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96" podUID="b330ed87-c300-412d-9ef5-91c4460e4794" containerName="istio-proxy" containerID="cri-o://cc255eca003a3ef3753ba00c2f2faaadef9a02c4a0ed2d926235a2cd6f5a4800" gracePeriod=30
Nov 01 07:23:56.339312 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:23:56.339206    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:24:01.608623 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.608559    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:24:01.690085 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.690049    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-credential-socket\") pod \"b330ed87-c300-412d-9ef5-91c4460e4794\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") "
Nov 01 07:24:01.690306 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.690112    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-workload-certs\") pod \"b330ed87-c300-412d-9ef5-91c4460e4794\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") "
Nov 01 07:24:01.690306 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.690164    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-workload-socket\") pod \"b330ed87-c300-412d-9ef5-91c4460e4794\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") "
Nov 01 07:24:01.690306 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.690200    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-istio-data\") pod \"b330ed87-c300-412d-9ef5-91c4460e4794\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") "
Nov 01 07:24:01.690306 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.690259    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b330ed87-c300-412d-9ef5-91c4460e4794-kube-api-access-txcrq\" (UniqueName: \"kubernetes.io/projected/b330ed87-c300-412d-9ef5-91c4460e4794-kube-api-access-txcrq\") pod \"b330ed87-c300-412d-9ef5-91c4460e4794\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") "
Nov 01 07:24:01.690306 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.690299    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b330ed87-c300-412d-9ef5-91c4460e4794-istio-token\" (UniqueName: \"kubernetes.io/projected/b330ed87-c300-412d-9ef5-91c4460e4794-istio-token\") pod \"b330ed87-c300-412d-9ef5-91c4460e4794\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") "
Nov 01 07:24:01.690643 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.690360    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b330ed87-c300-412d-9ef5-91c4460e4794-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b330ed87-c300-412d-9ef5-91c4460e4794-istio-podinfo\") pod \"b330ed87-c300-412d-9ef5-91c4460e4794\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") "
Nov 01 07:24:01.690643 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.690383    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-credential-socket" pod "b330ed87-c300-412d-9ef5-91c4460e4794" (UID: "b330ed87-c300-412d-9ef5-91c4460e4794"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:24:01.690643 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.690437    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-istio-envoy\") pod \"b330ed87-c300-412d-9ef5-91c4460e4794\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") "
Nov 01 07:24:01.690643 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.690479    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b330ed87-c300-412d-9ef5-91c4460e4794-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b330ed87-c300-412d-9ef5-91c4460e4794-istiod-ca-cert\") pod \"b330ed87-c300-412d-9ef5-91c4460e4794\" (UID: \"b330ed87-c300-412d-9ef5-91c4460e4794\") "
Nov 01 07:24:01.690643 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.690565    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-workload-certs" pod "b330ed87-c300-412d-9ef5-91c4460e4794" (UID: "b330ed87-c300-412d-9ef5-91c4460e4794"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:24:01.690643 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.690586    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-workload-socket" pod "b330ed87-c300-412d-9ef5-91c4460e4794" (UID: "b330ed87-c300-412d-9ef5-91c4460e4794"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:24:01.690951 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.690661    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-istio-data" pod "b330ed87-c300-412d-9ef5-91c4460e4794" (UID: "b330ed87-c300-412d-9ef5-91c4460e4794"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:24:01.690951 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.690923    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b330ed87-c300-412d-9ef5-91c4460e4794-istiod-ca-cert" pod "b330ed87-c300-412d-9ef5-91c4460e4794" (UID: "b330ed87-c300-412d-9ef5-91c4460e4794"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:24:01.691074 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.690992    5619 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-credential-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:01.691074 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.691006    5619 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-workload-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:01.691074 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.691015    5619 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-workload-socket\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:01.691074 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.691027    5619 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-istio-data\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:01.692662 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.692635    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b330ed87-c300-412d-9ef5-91c4460e4794-istio-token" pod "b330ed87-c300-412d-9ef5-91c4460e4794" (UID: "b330ed87-c300-412d-9ef5-91c4460e4794"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:24:01.692775 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.692669    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b330ed87-c300-412d-9ef5-91c4460e4794-kube-api-access-txcrq" pod "b330ed87-c300-412d-9ef5-91c4460e4794" (UID: "b330ed87-c300-412d-9ef5-91c4460e4794"). InnerVolumeSpecName "kube-api-access-txcrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:24:01.692920 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.692891    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b330ed87-c300-412d-9ef5-91c4460e4794-istio-podinfo" pod "b330ed87-c300-412d-9ef5-91c4460e4794" (UID: "b330ed87-c300-412d-9ef5-91c4460e4794"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 07:24:01.692920 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.692896    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-istio-envoy" pod "b330ed87-c300-412d-9ef5-91c4460e4794" (UID: "b330ed87-c300-412d-9ef5-91c4460e4794"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 07:24:01.792216 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.792178    5619 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b330ed87-c300-412d-9ef5-91c4460e4794-istio-token\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:01.792216 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.792220    5619 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b330ed87-c300-412d-9ef5-91c4460e4794-istio-podinfo\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:01.792509 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.792236    5619 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b330ed87-c300-412d-9ef5-91c4460e4794-istio-envoy\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:01.792509 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.792249    5619 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b330ed87-c300-412d-9ef5-91c4460e4794-istiod-ca-cert\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:01.792509 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:01.792265    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txcrq\" (UniqueName: \"kubernetes.io/projected/b330ed87-c300-412d-9ef5-91c4460e4794-kube-api-access-txcrq\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:02.179122 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:02.179087    5619 generic.go:356] "Generic (PLEG): container finished" podID="b330ed87-c300-412d-9ef5-91c4460e4794" containerID="cc255eca003a3ef3753ba00c2f2faaadef9a02c4a0ed2d926235a2cd6f5a4800" exitCode=0
Nov 01 07:24:02.179305 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:02.179136    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"
Nov 01 07:24:02.179305 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:02.179169    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96" event={"ID":"b330ed87-c300-412d-9ef5-91c4460e4794","Type":"ContainerDied","Data":"cc255eca003a3ef3753ba00c2f2faaadef9a02c4a0ed2d926235a2cd6f5a4800"}
Nov 01 07:24:02.179305 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:02.179225    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96" event={"ID":"b330ed87-c300-412d-9ef5-91c4460e4794","Type":"ContainerDied","Data":"46620e24b212098a9cd836355e9bb06f8469d06595413979cc1374a7c25103d5"}
Nov 01 07:24:02.179305 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:02.179247    5619 scope.go:122] "RemoveContainer" containerID="cc255eca003a3ef3753ba00c2f2faaadef9a02c4a0ed2d926235a2cd6f5a4800"
Nov 01 07:24:02.189882 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:02.189866    5619 scope.go:122] "RemoveContainer" containerID="7631a0dea019cf3c7b9afc91b27d471c50aaaade191953ca5adce1c098cdb81b"
Nov 01 07:24:02.204942 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:02.204915    5619 scope.go:122] "RemoveContainer" containerID="cc255eca003a3ef3753ba00c2f2faaadef9a02c4a0ed2d926235a2cd6f5a4800"
Nov 01 07:24:02.205305 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:24:02.205285    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc255eca003a3ef3753ba00c2f2faaadef9a02c4a0ed2d926235a2cd6f5a4800\": container with ID starting with cc255eca003a3ef3753ba00c2f2faaadef9a02c4a0ed2d926235a2cd6f5a4800 not found: ID does not exist" containerID="cc255eca003a3ef3753ba00c2f2faaadef9a02c4a0ed2d926235a2cd6f5a4800"
Nov 01 07:24:02.205377 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:02.205342    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc255eca003a3ef3753ba00c2f2faaadef9a02c4a0ed2d926235a2cd6f5a4800"} err="failed to get container status \"cc255eca003a3ef3753ba00c2f2faaadef9a02c4a0ed2d926235a2cd6f5a4800\": rpc error: code = NotFound desc = could not find container \"cc255eca003a3ef3753ba00c2f2faaadef9a02c4a0ed2d926235a2cd6f5a4800\": container with ID starting with cc255eca003a3ef3753ba00c2f2faaadef9a02c4a0ed2d926235a2cd6f5a4800 not found: ID does not exist"
Nov 01 07:24:02.205377 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:02.205372    5619 scope.go:122] "RemoveContainer" containerID="7631a0dea019cf3c7b9afc91b27d471c50aaaade191953ca5adce1c098cdb81b"
Nov 01 07:24:02.205724 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:24:02.205687    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7631a0dea019cf3c7b9afc91b27d471c50aaaade191953ca5adce1c098cdb81b\": container with ID starting with 7631a0dea019cf3c7b9afc91b27d471c50aaaade191953ca5adce1c098cdb81b not found: ID does not exist" containerID="7631a0dea019cf3c7b9afc91b27d471c50aaaade191953ca5adce1c098cdb81b"
Nov 01 07:24:02.205824 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:02.205736    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7631a0dea019cf3c7b9afc91b27d471c50aaaade191953ca5adce1c098cdb81b"} err="failed to get container status \"7631a0dea019cf3c7b9afc91b27d471c50aaaade191953ca5adce1c098cdb81b\": rpc error: code = NotFound desc = could not find container \"7631a0dea019cf3c7b9afc91b27d471c50aaaade191953ca5adce1c098cdb81b\": container with ID starting with 7631a0dea019cf3c7b9afc91b27d471c50aaaade191953ca5adce1c098cdb81b not found: ID does not exist"
Nov 01 07:24:02.346476 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:02.346435    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"]
Nov 01 07:24:02.397376 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:02.397342    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c6c6bf-openshift-default-676795c8f7v7l96"]
Nov 01 07:24:02.962690 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:02.962651    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b330ed87-c300-412d-9ef5-91c4460e4794" path="/var/lib/kubelet/pods/b330ed87-c300-412d-9ef5-91c4460e4794/volumes"
Nov 01 07:24:16.099342 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:16.099293    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:24:25.879033 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:24:25.878998    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:24:34.397698 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.397657    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:24:34.540923 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.540888    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:24:34.775370 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.775259    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-66f65f7fd7-mhshn"]
Nov 01 07:24:34.775716 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.775694    5619 cpu_manager.go:414] "containerMap: removing container" podUID="7cc0309f-a41e-4497-a576-3422db4b8ba9" containerName="maas-api"
Nov 01 07:24:34.775716 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.775713    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc0309f-a41e-4497-a576-3422db4b8ba9" containerName="maas-api"
Nov 01 07:24:34.775954 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.775749    5619 cpu_manager.go:414] "containerMap: removing container" podUID="b330ed87-c300-412d-9ef5-91c4460e4794" containerName="istio-proxy"
Nov 01 07:24:34.775954 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.775754    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b330ed87-c300-412d-9ef5-91c4460e4794" containerName="istio-proxy"
Nov 01 07:24:34.775954 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.775765    5619 cpu_manager.go:414] "containerMap: removing container" podUID="b330ed87-c300-412d-9ef5-91c4460e4794" containerName="istio-proxy"
Nov 01 07:24:34.775954 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.775771    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b330ed87-c300-412d-9ef5-91c4460e4794" containerName="istio-proxy"
Nov 01 07:24:34.775954 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.775856    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc0309f-a41e-4497-a576-3422db4b8ba9" containerName="maas-api"
Nov 01 07:24:34.775954 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.775877    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="b330ed87-c300-412d-9ef5-91c4460e4794" containerName="istio-proxy"
Nov 01 07:24:34.775954 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.775887    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="b330ed87-c300-412d-9ef5-91c4460e4794" containerName="istio-proxy"
Nov 01 07:24:34.779200 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.779180    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-mhshn"
Nov 01 07:24:34.854761 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.854729    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31dd79d6-70ea-471b-9557-4c542c62aaf6-openshift-service-ca\") pod \"maas-api-66f65f7fd7-mhshn\" (UID: \"31dd79d6-70ea-471b-9557-4c542c62aaf6\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-mhshn"
Nov 01 07:24:34.854934 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.854790    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31dd79d6-70ea-471b-9557-4c542c62aaf6-metrics-certs\") pod \"maas-api-66f65f7fd7-mhshn\" (UID: \"31dd79d6-70ea-471b-9557-4c542c62aaf6\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-mhshn"
Nov 01 07:24:34.854934 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.854888    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69mtb\" (UniqueName: \"kubernetes.io/projected/31dd79d6-70ea-471b-9557-4c542c62aaf6-kube-api-access-69mtb\") pod \"maas-api-66f65f7fd7-mhshn\" (UID: \"31dd79d6-70ea-471b-9557-4c542c62aaf6\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-mhshn"
Nov 01 07:24:34.855016 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.854942    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31dd79d6-70ea-471b-9557-4c542c62aaf6-maas-api-tls\") pod \"maas-api-66f65f7fd7-mhshn\" (UID: \"31dd79d6-70ea-471b-9557-4c542c62aaf6\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-mhshn"
Nov 01 07:24:34.885864 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.885830    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66f65f7fd7-mhshn"]
Nov 01 07:24:34.955730 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.955697    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31dd79d6-70ea-471b-9557-4c542c62aaf6-maas-api-tls\") pod \"maas-api-66f65f7fd7-mhshn\" (UID: \"31dd79d6-70ea-471b-9557-4c542c62aaf6\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-mhshn"
Nov 01 07:24:34.955901 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.955803    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31dd79d6-70ea-471b-9557-4c542c62aaf6-openshift-service-ca\") pod \"maas-api-66f65f7fd7-mhshn\" (UID: \"31dd79d6-70ea-471b-9557-4c542c62aaf6\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-mhshn"
Nov 01 07:24:34.955901 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.955873    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31dd79d6-70ea-471b-9557-4c542c62aaf6-metrics-certs\") pod \"maas-api-66f65f7fd7-mhshn\" (UID: \"31dd79d6-70ea-471b-9557-4c542c62aaf6\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-mhshn"
Nov 01 07:24:34.956005 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.955929    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69mtb\" (UniqueName: \"kubernetes.io/projected/31dd79d6-70ea-471b-9557-4c542c62aaf6-kube-api-access-69mtb\") pod \"maas-api-66f65f7fd7-mhshn\" (UID: \"31dd79d6-70ea-471b-9557-4c542c62aaf6\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-mhshn"
Nov 01 07:24:34.956580 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.956542    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31dd79d6-70ea-471b-9557-4c542c62aaf6-openshift-service-ca\") pod \"maas-api-66f65f7fd7-mhshn\" (UID: \"31dd79d6-70ea-471b-9557-4c542c62aaf6\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-mhshn"
Nov 01 07:24:34.958396 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.958372    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31dd79d6-70ea-471b-9557-4c542c62aaf6-metrics-certs\") pod \"maas-api-66f65f7fd7-mhshn\" (UID: \"31dd79d6-70ea-471b-9557-4c542c62aaf6\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-mhshn"
Nov 01 07:24:34.958552 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:34.958453    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31dd79d6-70ea-471b-9557-4c542c62aaf6-maas-api-tls\") pod \"maas-api-66f65f7fd7-mhshn\" (UID: \"31dd79d6-70ea-471b-9557-4c542c62aaf6\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-mhshn"
Nov 01 07:24:35.017177 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:35.017144    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69mtb\" (UniqueName: \"kubernetes.io/projected/31dd79d6-70ea-471b-9557-4c542c62aaf6-kube-api-access-69mtb\") pod \"maas-api-66f65f7fd7-mhshn\" (UID: \"31dd79d6-70ea-471b-9557-4c542c62aaf6\") " pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-mhshn"
Nov 01 07:24:35.093405 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:35.093376    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-mhshn"
Nov 01 07:24:35.466452 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:35.466426    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66f65f7fd7-mhshn"]
Nov 01 07:24:35.468287 ip-10-0-129-213 kubenswrapper[5619]: W1101 07:24:35.468260    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31dd79d6_70ea_471b_9557_4c542c62aaf6.slice/crio-c3985e8e21d39328be38fea2b1c87384431c9c53ed64df707ac1f10c47f01bf1 WatchSource:0}: Error finding container c3985e8e21d39328be38fea2b1c87384431c9c53ed64df707ac1f10c47f01bf1: Status 404 returned error can't find the container with id c3985e8e21d39328be38fea2b1c87384431c9c53ed64df707ac1f10c47f01bf1
Nov 01 07:24:35.884954 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:35.884913    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:24:36.073544 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:36.073513    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:24:36.357994 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:36.357963    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-mhshn" event={"ID":"31dd79d6-70ea-471b-9557-4c542c62aaf6","Type":"ContainerStarted","Data":"c3985e8e21d39328be38fea2b1c87384431c9c53ed64df707ac1f10c47f01bf1"}
Nov 01 07:24:37.364999 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:37.364961    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-mhshn" event={"ID":"31dd79d6-70ea-471b-9557-4c542c62aaf6","Type":"ContainerStarted","Data":"9116fd0cc6cd6ffed51f7484a80460f0bb2d6a4dc1b2586aaeda53a0e307cc35"}
Nov 01 07:24:37.365442 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:37.365051    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-mhshn"
Nov 01 07:24:37.484982 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:37.484930    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-mhshn" podStartSLOduration=2.604411111 podStartE2EDuration="3.484915934s" totalImagesPullingTime="880.504823ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:24:37.484055777 +0000 UTC m=+5914.694879288" watchObservedRunningTime="2026-11-01 07:24:37.484915934 +0000 UTC m=+5914.695739442"
Nov 01 07:24:43.375908 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:43.375877    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-66f65f7fd7-mhshn"
Nov 01 07:24:43.548157 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:43.548125    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"]
Nov 01 07:24:43.548413 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:43.548388    5619 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws" podUID="c0ecdd10-e2e4-4956-a985-741f86e7f3a6" containerName="maas-api" containerID="cri-o://80bd178d909dc05fd46b27959aba6e99a6f2e4bc6f863511f6095b271722fc98" gracePeriod=30
Nov 01 07:24:43.804606 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:43.804517    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws" podUID="c0ecdd10-e2e4-4956-a985-741f86e7f3a6" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.31:8443/health\": dial tcp 10.133.0.31:8443: connect: connection refused"
Nov 01 07:24:44.799855 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:44.799829    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"
Nov 01 07:24:44.855237 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:44.855203    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-metrics-certs\") pod \"c0ecdd10-e2e4-4956-a985-741f86e7f3a6\" (UID: \"c0ecdd10-e2e4-4956-a985-741f86e7f3a6\") "
Nov 01 07:24:44.855417 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:44.855291    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-kube-api-access-wqvmt\" (UniqueName: \"kubernetes.io/projected/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-kube-api-access-wqvmt\") pod \"c0ecdd10-e2e4-4956-a985-741f86e7f3a6\" (UID: \"c0ecdd10-e2e4-4956-a985-741f86e7f3a6\") "
Nov 01 07:24:44.855417 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:44.855388    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-openshift-service-ca\") pod \"c0ecdd10-e2e4-4956-a985-741f86e7f3a6\" (UID: \"c0ecdd10-e2e4-4956-a985-741f86e7f3a6\") "
Nov 01 07:24:44.855553 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:44.855446    5619 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-maas-api-tls\") pod \"c0ecdd10-e2e4-4956-a985-741f86e7f3a6\" (UID: \"c0ecdd10-e2e4-4956-a985-741f86e7f3a6\") "
Nov 01 07:24:44.855752 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:44.855726    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-openshift-service-ca" pod "c0ecdd10-e2e4-4956-a985-741f86e7f3a6" (UID: "c0ecdd10-e2e4-4956-a985-741f86e7f3a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 07:24:44.857561 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:44.857531    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-maas-api-tls" pod "c0ecdd10-e2e4-4956-a985-741f86e7f3a6" (UID: "c0ecdd10-e2e4-4956-a985-741f86e7f3a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:24:44.857561 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:44.857549    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-metrics-certs" pod "c0ecdd10-e2e4-4956-a985-741f86e7f3a6" (UID: "c0ecdd10-e2e4-4956-a985-741f86e7f3a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 07:24:44.857691 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:44.857545    5619 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-kube-api-access-wqvmt" pod "c0ecdd10-e2e4-4956-a985-741f86e7f3a6" (UID: "c0ecdd10-e2e4-4956-a985-741f86e7f3a6"). InnerVolumeSpecName "kube-api-access-wqvmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 07:24:44.956087 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:44.956058    5619 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-metrics-certs\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:44.956087 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:44.956085    5619 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqvmt\" (UniqueName: \"kubernetes.io/projected/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-kube-api-access-wqvmt\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:44.956298 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:44.956096    5619 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-openshift-service-ca\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:44.956298 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:44.956106    5619 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0ecdd10-e2e4-4956-a985-741f86e7f3a6-maas-api-tls\") on node \"ip-10-0-129-213.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 07:24:45.405739 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:45.405704    5619 generic.go:356] "Generic (PLEG): container finished" podID="c0ecdd10-e2e4-4956-a985-741f86e7f3a6" containerID="80bd178d909dc05fd46b27959aba6e99a6f2e4bc6f863511f6095b271722fc98" exitCode=0
Nov 01 07:24:45.405913 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:45.405751    5619 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"
Nov 01 07:24:45.405913 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:45.405757    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws" event={"ID":"c0ecdd10-e2e4-4956-a985-741f86e7f3a6","Type":"ContainerDied","Data":"80bd178d909dc05fd46b27959aba6e99a6f2e4bc6f863511f6095b271722fc98"}
Nov 01 07:24:45.405913 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:45.405809    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bdb66f487-jxkws" event={"ID":"c0ecdd10-e2e4-4956-a985-741f86e7f3a6","Type":"ContainerDied","Data":"76bafb82de183bb8814a3d0f0d9378ca035fddacac71398340916423bbb1c959"}
Nov 01 07:24:45.405913 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:45.405839    5619 scope.go:122] "RemoveContainer" containerID="80bd178d909dc05fd46b27959aba6e99a6f2e4bc6f863511f6095b271722fc98"
Nov 01 07:24:45.414412 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:45.414392    5619 scope.go:122] "RemoveContainer" containerID="80bd178d909dc05fd46b27959aba6e99a6f2e4bc6f863511f6095b271722fc98"
Nov 01 07:24:45.414716 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:24:45.414689    5619 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80bd178d909dc05fd46b27959aba6e99a6f2e4bc6f863511f6095b271722fc98\": container with ID starting with 80bd178d909dc05fd46b27959aba6e99a6f2e4bc6f863511f6095b271722fc98 not found: ID does not exist" containerID="80bd178d909dc05fd46b27959aba6e99a6f2e4bc6f863511f6095b271722fc98"
Nov 01 07:24:45.414783 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:45.414727    5619 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80bd178d909dc05fd46b27959aba6e99a6f2e4bc6f863511f6095b271722fc98"} err="failed to get container status \"80bd178d909dc05fd46b27959aba6e99a6f2e4bc6f863511f6095b271722fc98\": rpc error: code = NotFound desc = could not find container \"80bd178d909dc05fd46b27959aba6e99a6f2e4bc6f863511f6095b271722fc98\": container with ID starting with 80bd178d909dc05fd46b27959aba6e99a6f2e4bc6f863511f6095b271722fc98 not found: ID does not exist"
Nov 01 07:24:45.572587 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:45.572558    5619 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"]
Nov 01 07:24:45.623356 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:45.623314    5619 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-bdb66f487-jxkws"]
Nov 01 07:24:45.942867 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:45.942833    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:24:46.960456 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:46.960424    5619 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ecdd10-e2e4-4956-a985-741f86e7f3a6" path="/var/lib/kubelet/pods/c0ecdd10-e2e4-4956-a985-741f86e7f3a6/volumes"
Nov 01 07:24:47.931020 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:47.930973    5619 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal" podUID="ef0f634a7979920e9df606f3bb474aac" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:24:50.452121 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:24:50.452084    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:24:55.878765 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:24:55.878726    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:25:25.878165 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:25:25.878127    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:25:46.302469 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:25:46.302427    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:25:50.763930 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:25:50.763895    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:25:55.878562 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:25:55.878521    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:26:25.882286 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:26:25.882246    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:26:27.255398 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:26:27.250835    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:26:27.444583 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:26:27.444548    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:26:46.699735 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:26:46.699697    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:26:51.132739 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:26:51.132698    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:26:55.879183 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:26:55.879145    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:27:19.212344 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:27:19.211454    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-qrvtb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:27:19.212344 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:27:19.211560    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-qrvtb" podUID="ac4eb8e8-a937-4744-9e9a-8b8f65803484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:27:19.234475 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:27:19.234441    5619 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-6sjbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 07:27:19.234623 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:27:19.234495    5619 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-6sjbq" podUID="f706dfe8-5cf7-42c5-855e-85f021da7b96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 07:27:25.880863 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:27:25.880818    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:27:46.998404 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:27:46.998363    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:27:51.481459 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:27:51.481428    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:27:55.886449 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:27:55.886402    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:28:25.879737 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:28:25.879670    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:28:33.637365 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:28:33.637299    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:28:47.352522 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:28:47.352489    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:28:51.030331 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:28:51.030287    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:28:51.826159 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:28:51.826123    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:28:55.878778 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:28:55.878740    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:28:56.785422 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:28:56.785385    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-zgwzb_02fae5ae-abd3-4895-a525-42ff3bae5ff4/manager/0.log"
Nov 01 07:28:57.624486 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:28:57.624456    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bf8b5db8-w9m7j_a322e32c-aaf3-448f-8fbd-49459435c6b5/manager/0.log"
Nov 01 07:28:57.875910 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:28:57.875836    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-zvcc2_9d763214-1e93-47b0-bfb4-e3321fc6d742/server/0.log"
Nov 01 07:29:05.325574 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:29:05.325541    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:29:10.037922 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:29:10.037888    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-66f65f7fd7-mhshn_31dd79d6-70ea-471b-9557-4c542c62aaf6/maas-api/0.log"
Nov 01 07:29:22.113899 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:29:22.113867    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-6fqw4_8745c5be-e512-41f6-a186-8211cac046ee/postgres/0.log"
Nov 01 07:29:25.302852 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:29:25.302820    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hw2bp_2a0527b8-6adb-4ed9-a183-8263f8cbbb0d/manager/0.log"
Nov 01 07:29:25.877975 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:29:25.877932    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:29:27.540148 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:29:27.540093    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:29:28.367380 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:29:28.367345    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:29:36.860073 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:29:36.860039    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c557tpltp_41c04bed-9587-4bd7-904c-cc0846d94e86/istio-proxy/0.log"
Nov 01 07:29:37.644112 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:29:37.644080    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84b94dcd67-zxvcc_e8797e16-282e-476e-8008-92456f432069/kube-auth-proxy/0.log"
Nov 01 07:29:38.117280 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:29:38.117255    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-6sjbq_f706dfe8-5cf7-42c5-855e-85f021da7b96/istio-proxy/0.log"
Nov 01 07:29:39.554008 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:29:39.553970    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-qrvtb_ac4eb8e8-a937-4744-9e9a-8b8f65803484/istio-proxy/0.log"
Nov 01 07:29:44.763731 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:29:44.763634    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:29:44.903507 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:29:44.903474    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:29:47.691214 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:29:47.691155    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:29:50.011821 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:29:50.011789    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-x9mzj_d299e7a4-892c-404f-8346-91bcae60326e/istio-proxy/0.log"
Nov 01 07:29:50.383413 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:29:50.383381    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl_45899a85-a566-4cff-be6e-a2b0d6c7afed/istio-proxy/0.log"
Nov 01 07:29:52.084860 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:29:52.084825    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-nb56f_1951f4ec-2955-4b93-a40e-34de8400b217/main/0.log"
Nov 01 07:29:52.176616 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:29:52.176558    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:29:52.812056 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:29:52.812029    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-jvlvd_305429ba-4531-4538-86e2-34e1a9387083/main/0.log"
Nov 01 07:29:53.302269 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:29:53.302238    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-npc9b_25955c22-ca47-4d49-8253-0f9660b74633/main/0.log"
Nov 01 07:29:55.878401 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:29:55.878357    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:30:20.380487 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:20.380447    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:30:21.217873 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:21.217836    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:30:25.878760 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:30:25.878719    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:30:48.079510 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:48.079462    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:30:52.511807 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:52.511771    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:30:53.781070 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:53.781034    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9jzqt/must-gather-m7nfq"]
Nov 01 07:30:53.781591 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:53.781396    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:30:53.781657 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:53.781611    5619 cpu_manager.go:414] "containerMap: removing container" podUID="c0ecdd10-e2e4-4956-a985-741f86e7f3a6" containerName="maas-api"
Nov 01 07:30:53.781657 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:53.781630    5619 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ecdd10-e2e4-4956-a985-741f86e7f3a6" containerName="maas-api"
Nov 01 07:30:53.781752 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:53.781733    5619 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ecdd10-e2e4-4956-a985-741f86e7f3a6" containerName="maas-api"
Nov 01 07:30:53.785004 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:53.784989    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9jzqt/must-gather-m7nfq"
Nov 01 07:30:53.821748 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:53.821721    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx5kz\" (UniqueName: \"kubernetes.io/projected/13ac1e05-5280-4b23-b1a5-65ce0164cc25-kube-api-access-sx5kz\") pod \"must-gather-m7nfq\" (UID: \"13ac1e05-5280-4b23-b1a5-65ce0164cc25\") " pod="openshift-must-gather-9jzqt/must-gather-m7nfq"
Nov 01 07:30:53.821880 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:53.821775    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/13ac1e05-5280-4b23-b1a5-65ce0164cc25-must-gather-output\") pod \"must-gather-m7nfq\" (UID: \"13ac1e05-5280-4b23-b1a5-65ce0164cc25\") " pod="openshift-must-gather-9jzqt/must-gather-m7nfq"
Nov 01 07:30:53.891085 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:53.891053    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9jzqt\"/\"kube-root-ca.crt\""
Nov 01 07:30:53.891265 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:53.891230    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9jzqt\"/\"openshift-service-ca.crt\""
Nov 01 07:30:53.891356 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:53.891318    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9jzqt\"/\"default-dockercfg-ps2dr\""
Nov 01 07:30:53.892921 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:53.892902    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9jzqt/must-gather-m7nfq"]
Nov 01 07:30:53.923156 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:53.923111    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx5kz\" (UniqueName: \"kubernetes.io/projected/13ac1e05-5280-4b23-b1a5-65ce0164cc25-kube-api-access-sx5kz\") pod \"must-gather-m7nfq\" (UID: \"13ac1e05-5280-4b23-b1a5-65ce0164cc25\") " pod="openshift-must-gather-9jzqt/must-gather-m7nfq"
Nov 01 07:30:53.923372 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:53.923178    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/13ac1e05-5280-4b23-b1a5-65ce0164cc25-must-gather-output\") pod \"must-gather-m7nfq\" (UID: \"13ac1e05-5280-4b23-b1a5-65ce0164cc25\") " pod="openshift-must-gather-9jzqt/must-gather-m7nfq"
Nov 01 07:30:53.923584 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:53.923565    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/13ac1e05-5280-4b23-b1a5-65ce0164cc25-must-gather-output\") pod \"must-gather-m7nfq\" (UID: \"13ac1e05-5280-4b23-b1a5-65ce0164cc25\") " pod="openshift-must-gather-9jzqt/must-gather-m7nfq"
Nov 01 07:30:53.976514 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:53.976466    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:30:53.984869 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:53.984838    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx5kz\" (UniqueName: \"kubernetes.io/projected/13ac1e05-5280-4b23-b1a5-65ce0164cc25-kube-api-access-sx5kz\") pod \"must-gather-m7nfq\" (UID: \"13ac1e05-5280-4b23-b1a5-65ce0164cc25\") " pod="openshift-must-gather-9jzqt/must-gather-m7nfq"
Nov 01 07:30:54.098888 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:54.098854    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9jzqt/must-gather-m7nfq"
Nov 01 07:30:54.470911 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:54.470885    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9jzqt/must-gather-m7nfq"]
Nov 01 07:30:54.473107 ip-10-0-129-213 kubenswrapper[5619]: W1101 07:30:54.473081    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13ac1e05_5280_4b23_b1a5_65ce0164cc25.slice/crio-9c745347877af2522609a9462107b4f1680313a8c5367663646df7354ba93a96 WatchSource:0}: Error finding container 9c745347877af2522609a9462107b4f1680313a8c5367663646df7354ba93a96: Status 404 returned error can't find the container with id 9c745347877af2522609a9462107b4f1680313a8c5367663646df7354ba93a96
Nov 01 07:30:54.475108 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:54.475082    5619 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 07:30:55.346590 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:30:55.346557    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9jzqt/must-gather-m7nfq" event={"ID":"13ac1e05-5280-4b23-b1a5-65ce0164cc25","Type":"ContainerStarted","Data":"9c745347877af2522609a9462107b4f1680313a8c5367663646df7354ba93a96"}
Nov 01 07:30:55.878809 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:30:55.878775    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:31:01.380598 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:01.380561    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9jzqt/must-gather-m7nfq" event={"ID":"13ac1e05-5280-4b23-b1a5-65ce0164cc25","Type":"ContainerStarted","Data":"329e74c4ac8dabdbeb5182c260547bee2986fbbfb419a5fc6f08a14aaa0bb5d7"}
Nov 01 07:31:01.381001 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:01.380608    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9jzqt/must-gather-m7nfq" event={"ID":"13ac1e05-5280-4b23-b1a5-65ce0164cc25","Type":"ContainerStarted","Data":"69dbfee720a6c86f05990f0a0010eafe2133e173f7b7712080a89b2f536cf4ae"}
Nov 01 07:31:01.498770 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:01.498115    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9jzqt/must-gather-m7nfq" podStartSLOduration=2.353719206 podStartE2EDuration="8.498091467s" totalImagesPullingTime="6.144372261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:31:01.496949797 +0000 UTC m=+6298.707773307" watchObservedRunningTime="2026-11-01 07:31:01.498091467 +0000 UTC m=+6298.708914977"
Nov 01 07:31:03.802126 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:03.802081    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:31:03.948215 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:03.948177    5619 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log" limit=1024
Nov 01 07:31:12.565286 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:12.565200    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jh5gk_7f767e63-7cc7-4b5e-9941-860fe5f4cd62/global-pull-secret-syncer/0.log"
Nov 01 07:31:12.897524 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:12.897493    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-zgwzb_02fae5ae-abd3-4895-a525-42ff3bae5ff4/manager/0.log"
Nov 01 07:31:14.421842 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:14.421807    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bf8b5db8-w9m7j_a322e32c-aaf3-448f-8fbd-49459435c6b5/manager/0.log"
Nov 01 07:31:14.529216 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:14.529188    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-zvcc2_9d763214-1e93-47b0-bfb4-e3321fc6d742/server/0.log"
Nov 01 07:31:15.937562 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:15.937523    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4lm7r_05e09ea1-324b-49ef-a4a5-2645b50232a4/konnectivity-agent/0.log"
Nov 01 07:31:17.153968 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:17.153937    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-213.us-west-1.compute.internal_ef0f634a7979920e9df606f3bb474aac/haproxy/0.log"
Nov 01 07:31:19.563815 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:19.563785    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-zgwzb_02fae5ae-abd3-4895-a525-42ff3bae5ff4/manager/0.log"
Nov 01 07:31:20.616356 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:20.616311    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bf8b5db8-w9m7j_a322e32c-aaf3-448f-8fbd-49459435c6b5/manager/0.log"
Nov 01 07:31:20.725760 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:20.725721    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-zvcc2_9d763214-1e93-47b0-bfb4-e3321fc6d742/server/0.log"
Nov 01 07:31:23.545909 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:23.545876    5619 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv"]
Nov 01 07:31:23.549775 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:23.549756    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv"
Nov 01 07:31:23.604524 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:23.604489    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s25q4\"/\"openshift-service-ca.crt\""
Nov 01 07:31:23.604700 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:23.604594    5619 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s25q4\"/\"kube-root-ca.crt\""
Nov 01 07:31:23.626144 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:23.626115    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3f592253-d1c8-44d5-aeb5-371423a124e9-podres\") pod \"perf-node-gather-daemonset-2qwvv\" (UID: \"3f592253-d1c8-44d5-aeb5-371423a124e9\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv"
Nov 01 07:31:23.626291 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:23.626214    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f592253-d1c8-44d5-aeb5-371423a124e9-lib-modules\") pod \"perf-node-gather-daemonset-2qwvv\" (UID: \"3f592253-d1c8-44d5-aeb5-371423a124e9\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv"
Nov 01 07:31:23.626291 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:23.626254    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3f592253-d1c8-44d5-aeb5-371423a124e9-proc\") pod \"perf-node-gather-daemonset-2qwvv\" (UID: \"3f592253-d1c8-44d5-aeb5-371423a124e9\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv"
Nov 01 07:31:23.626403 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:23.626379    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f592253-d1c8-44d5-aeb5-371423a124e9-sys\") pod \"perf-node-gather-daemonset-2qwvv\" (UID: \"3f592253-d1c8-44d5-aeb5-371423a124e9\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv"
Nov 01 07:31:23.626444 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:23.626407    5619 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79cqz\" (UniqueName: \"kubernetes.io/projected/3f592253-d1c8-44d5-aeb5-371423a124e9-kube-api-access-79cqz\") pod \"perf-node-gather-daemonset-2qwvv\" (UID: \"3f592253-d1c8-44d5-aeb5-371423a124e9\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv"
Nov 01 07:31:23.655281 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:23.655259    5619 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s25q4\"/\"default-dockercfg-9wmvv\""
Nov 01 07:31:23.656991 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:23.656970    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv"]
Nov 01 07:31:23.727005 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:23.726973    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3f592253-d1c8-44d5-aeb5-371423a124e9-proc\") pod \"perf-node-gather-daemonset-2qwvv\" (UID: \"3f592253-d1c8-44d5-aeb5-371423a124e9\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv"
Nov 01 07:31:23.727200 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:23.727028    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f592253-d1c8-44d5-aeb5-371423a124e9-sys\") pod \"perf-node-gather-daemonset-2qwvv\" (UID: \"3f592253-d1c8-44d5-aeb5-371423a124e9\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv"
Nov 01 07:31:23.727200 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:23.727059    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79cqz\" (UniqueName: \"kubernetes.io/projected/3f592253-d1c8-44d5-aeb5-371423a124e9-kube-api-access-79cqz\") pod \"perf-node-gather-daemonset-2qwvv\" (UID: \"3f592253-d1c8-44d5-aeb5-371423a124e9\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv"
Nov 01 07:31:23.727200 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:23.727113    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3f592253-d1c8-44d5-aeb5-371423a124e9-proc\") pod \"perf-node-gather-daemonset-2qwvv\" (UID: \"3f592253-d1c8-44d5-aeb5-371423a124e9\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv"
Nov 01 07:31:23.727200 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:23.727181    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f592253-d1c8-44d5-aeb5-371423a124e9-sys\") pod \"perf-node-gather-daemonset-2qwvv\" (UID: \"3f592253-d1c8-44d5-aeb5-371423a124e9\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv"
Nov 01 07:31:23.727441 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:23.727188    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3f592253-d1c8-44d5-aeb5-371423a124e9-podres\") pod \"perf-node-gather-daemonset-2qwvv\" (UID: \"3f592253-d1c8-44d5-aeb5-371423a124e9\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv"
Nov 01 07:31:23.727441 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:23.727235    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3f592253-d1c8-44d5-aeb5-371423a124e9-podres\") pod \"perf-node-gather-daemonset-2qwvv\" (UID: \"3f592253-d1c8-44d5-aeb5-371423a124e9\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv"
Nov 01 07:31:23.727441 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:23.727265    5619 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f592253-d1c8-44d5-aeb5-371423a124e9-lib-modules\") pod \"perf-node-gather-daemonset-2qwvv\" (UID: \"3f592253-d1c8-44d5-aeb5-371423a124e9\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv"
Nov 01 07:31:23.727441 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:23.727390    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f592253-d1c8-44d5-aeb5-371423a124e9-lib-modules\") pod \"perf-node-gather-daemonset-2qwvv\" (UID: \"3f592253-d1c8-44d5-aeb5-371423a124e9\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv"
Nov 01 07:31:23.789697 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:23.789669    5619 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79cqz\" (UniqueName: \"kubernetes.io/projected/3f592253-d1c8-44d5-aeb5-371423a124e9-kube-api-access-79cqz\") pod \"perf-node-gather-daemonset-2qwvv\" (UID: \"3f592253-d1c8-44d5-aeb5-371423a124e9\") " pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv"
Nov 01 07:31:23.864989 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:23.864959    5619 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv"
Nov 01 07:31:24.242164 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:24.242138    5619 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv"]
Nov 01 07:31:24.244081 ip-10-0-129-213 kubenswrapper[5619]: W1101 07:31:24.244055    5619 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3f592253_d1c8_44d5_aeb5_371423a124e9.slice/crio-1ed67234821acc2223466422d33cb60a29d9a36e19120c4a32d894f5af3fc5cf WatchSource:0}: Error finding container 1ed67234821acc2223466422d33cb60a29d9a36e19120c4a32d894f5af3fc5cf: Status 404 returned error can't find the container with id 1ed67234821acc2223466422d33cb60a29d9a36e19120c4a32d894f5af3fc5cf
Nov 01 07:31:24.498435 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:24.498352    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv" event={"ID":"3f592253-d1c8-44d5-aeb5-371423a124e9","Type":"ContainerStarted","Data":"32017bcf9abf339587e1d2d7adc68c1ca4fba7a41e1b878864381eaef8dc4e56"}
Nov 01 07:31:24.498435 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:24.498414    5619 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv" event={"ID":"3f592253-d1c8-44d5-aeb5-371423a124e9","Type":"ContainerStarted","Data":"1ed67234821acc2223466422d33cb60a29d9a36e19120c4a32d894f5af3fc5cf"}
Nov 01 07:31:24.498667 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:24.498533    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv"
Nov 01 07:31:24.618202 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:24.618143    5619 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv" podStartSLOduration=1.618123507 podStartE2EDuration="1.618123507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 07:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:31:24.616628188 +0000 UTC m=+6321.827451710" watchObservedRunningTime="2026-11-01 07:31:24.618123507 +0000 UTC m=+6321.828947016"
Nov 01 07:31:25.566350 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:25.566308    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hw2bp_2a0527b8-6adb-4ed9-a183-8263f8cbbb0d/manager/0.log"
Nov 01 07:31:25.763932 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:25.763899    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-zgwzb_02fae5ae-abd3-4895-a525-42ff3bae5ff4/manager/0.log"
Nov 01 07:31:25.879028 ip-10-0-129-213 kubenswrapper[5619]: E1101 07:31:25.878997    5619 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nt6ft" containerName="extractor"
Nov 01 07:31:26.783071 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:26.783009    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bf8b5db8-w9m7j_a322e32c-aaf3-448f-8fbd-49459435c6b5/manager/0.log"
Nov 01 07:31:26.891794 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:26.891762    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-zvcc2_9d763214-1e93-47b0-bfb4-e3321fc6d742/server/0.log"
Nov 01 07:31:29.011815 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:29.011781    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_63d18a9f-f582-4a00-a0b4-3d33f9363354/alertmanager/0.log"
Nov 01 07:31:29.428975 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:29.428943    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_63d18a9f-f582-4a00-a0b4-3d33f9363354/config-reloader/0.log"
Nov 01 07:31:29.848606 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:29.848575    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_63d18a9f-f582-4a00-a0b4-3d33f9363354/kube-rbac-proxy-web/0.log"
Nov 01 07:31:30.266021 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:30.265946    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_63d18a9f-f582-4a00-a0b4-3d33f9363354/kube-rbac-proxy/0.log"
Nov 01 07:31:30.514402 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:30.514374    5619 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s25q4/perf-node-gather-daemonset-2qwvv"
Nov 01 07:31:30.683883 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:30.683854    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_63d18a9f-f582-4a00-a0b4-3d33f9363354/kube-rbac-proxy-metric/0.log"
Nov 01 07:31:31.101689 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:31.101663    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_63d18a9f-f582-4a00-a0b4-3d33f9363354/prom-label-proxy/0.log"
Nov 01 07:31:31.508593 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:31.508516    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c557tpltp_41c04bed-9587-4bd7-904c-cc0846d94e86/istio-proxy/0.log"
Nov 01 07:31:31.519381 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:31.519359    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_63d18a9f-f582-4a00-a0b4-3d33f9363354/init-config-reloader/0.log"
Nov 01 07:31:32.346669 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:32.346638    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84b94dcd67-zxvcc_e8797e16-282e-476e-8008-92456f432069/kube-auth-proxy/0.log"
Nov 01 07:31:32.781303 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:32.781201    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-6sjbq_f706dfe8-5cf7-42c5-855e-85f021da7b96/istio-proxy/0.log"
Nov 01 07:31:33.617801 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:33.617773    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-69999ffccd-cvcrh_95fd5853-f502-4076-90a2-19fb99f52e2c/metrics-server/0.log"
Nov 01 07:31:34.058985 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:34.058885    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-qrvtb_ac4eb8e8-a937-4744-9e9a-8b8f65803484/istio-proxy/0.log"
Nov 01 07:31:34.069677 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:34.069649    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-2brlp_941ae7ae-e26d-4adb-8d74-729e4c038861/monitoring-plugin/0.log"
Nov 01 07:31:34.490267 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:34.490230    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-x9mzj_d299e7a4-892c-404f-8346-91bcae60326e/istio-proxy/0.log"
Nov 01 07:31:34.697174 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:34.697136    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-9b4747bc-jtjfl_45899a85-a566-4cff-be6e-a2b0d6c7afed/istio-proxy/0.log"
Nov 01 07:31:35.785039 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:35.785008    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m5vvc_ec642c5b-c364-4faa-94d5-bb5bf32a2cb8/node-exporter/0.log"
Nov 01 07:31:36.207863 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:36.207830    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m5vvc_ec642c5b-c364-4faa-94d5-bb5bf32a2cb8/kube-rbac-proxy/0.log"
Nov 01 07:31:36.625859 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:36.625827    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m5vvc_ec642c5b-c364-4faa-94d5-bb5bf32a2cb8/init-textfile/0.log"
Nov 01 07:31:39.582182 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:39.582146    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0d57ece9-d021-4395-b469-86a4efa4d857/prometheus/0.log"
Nov 01 07:31:40.000060 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:40.000027    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0d57ece9-d021-4395-b469-86a4efa4d857/config-reloader/0.log"
Nov 01 07:31:40.417994 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:40.417820    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0d57ece9-d021-4395-b469-86a4efa4d857/thanos-sidecar/0.log"
Nov 01 07:31:40.837537 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:40.837452    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0d57ece9-d021-4395-b469-86a4efa4d857/kube-rbac-proxy-web/0.log"
Nov 01 07:31:41.256232 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:41.256187    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0d57ece9-d021-4395-b469-86a4efa4d857/kube-rbac-proxy/0.log"
Nov 01 07:31:41.291211 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:41.291182    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hw2bp_2a0527b8-6adb-4ed9-a183-8263f8cbbb0d/manager/0.log"
Nov 01 07:31:41.676619 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:41.676586    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0d57ece9-d021-4395-b469-86a4efa4d857/kube-rbac-proxy-thanos/0.log"
Nov 01 07:31:42.097346 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:42.097285    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0d57ece9-d021-4395-b469-86a4efa4d857/init-config-reloader/0.log"
Nov 01 07:31:43.358784 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:43.358751    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-vwqd2_06f0eaf3-04d5-492f-9d3a-6b337091fbea/prometheus-operator-admission-webhook/0.log"
Nov 01 07:31:46.950062 ip-10-0-129-213 kubenswrapper[5619]: I1101 07:31:46.950025    5619 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c557tpltp_41c04bed-9587-4bd7-904c-cc0846d94e86/istio-proxy/0.log"